Re: [cmake-developers] [CMake] Integrating ExternalData with Artifactory

2014-09-22 Thread Brad King 
On 09/19/2014 06:07 PM, Taylor Braun-Jones wrote:
 On Fri, Sep 19, 2014 at 3:04 PM, Brad King wrote:
 I think it can be activated by a special format of an entry in
 ExternalData_URL_TEMPLATES that specifies a lookup key that maps
 to some kind of custom configuration of a .cmake script to include
 or command to launch with execute_process.
 
 How about defining new URL scheme like:
 
 externaldatacommand://*file|module*/*command*

Yes, something along these lines is what I had in mind.

 list(APPEND ExternalData_URL_TEMPLATES
   
 externaldatacommand://ArtifactoryExternalData/ARTIFACTORY_DOWNLOAD_FILE?ALGO=%(algo)
 )
 
 And inside Testing/CMake/ArtifactoryExternalData.cmake it would look like:
 
 function(ARTIFACTORY_DOWNLOAD_FILE file algo hash)

Since CMake does not support variable evaluation in a command name,
this would require configuring a file with code in it to launch for
each entry of this form in the url templates list.

Also CMAKE_MODULE_PATH will not be available to the ExternalData
module when it is launched at build time to do the download.  The
include() will have to work with a full path, perhaps stored in
the configured copy of ExternalData_config.cmake.in.  It could
be indexed with a key.  Perhaps:

  list(APPEND ExternalData_URL_TEMPLATES
ExternalDataCustomScript://MyFetch/%(algo)/%(hash)
)
  set(ExternalData_CUSTOM_SCRIPT_MyFetch /path/to/MyFetch.cmake)

The script would be include()-ed in place of the current call
to _ExternalData_download_file with the part of the URL after
the MyFetch/ already parsed into some variable(s).

-Brad

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] [CMake] Integrating ExternalData with Artifactory

2014-09-22 Thread Taylor Braun-Jones 
On Mon, Sep 22, 2014 at 10:07 AM, Brad King brad.k...@kitware.com wrote:

 Perhaps:

   list(APPEND ExternalData_URL_TEMPLATES
 ExternalDataCustomScript://MyFetch/%(algo)/%(hash)
 )
   set(ExternalData_CUSTOM_SCRIPT_MyFetch /path/to/MyFetch.cmake)

 The script would be include()-ed in place of the current call
 to _ExternalData_download_file with the part of the URL after
 the MyFetch/ already parsed into some variable(s).


This all sounds reasonable to me. I'd love to see this land in a future
CMake release. I'm willing to help if you need a tester/reviewer.
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

Re: [cmake-developers] [CMake] Integrating ExternalData with Artifactory

2014-09-19 Thread Taylor Braun-Jones 
Shifting discussion to cmake-developers.

On Fri, Sep 19, 2014 at 3:04 PM, Brad King brad.k...@kitware.com wrote:

 On 09/19/2014 12:23 PM, Taylor Braun-Jones wrote:
  integrate Artifactory with the ExternalData framework?
 
  the Artifactory REST API is a two step process

 I think it can be activated by a special format of an entry in
 ExternalData_URL_TEMPLATES that specifies a lookup key that maps
 to some kind of custom configuration of a .cmake script to include
 or command to launch with execute_process.


How about defining new URL scheme like:

externaldatacommand://*file|module*/*command*

Where *file|module* is something that can be passed to the include command
and defines a function or macro named *command*. *command* would then be
called like:

command(outputfile algo hash)

So, using Artifactory as an example case, a projects CMakeLists.txt might
look like:

list(APPEND CMAKE_MODULE_PATH
  ${CMAKE_SOURCE_DIR}/Testing/CMake
)
set(ARTIFACTORY_BASE_URL https://example.com/artifactory)
list(APPEND ExternalData_URL_TEMPLATES

externaldatacommand://ArtifactoryExternalData/ARTIFACTORY_DOWNLOAD_FILE?ALGO=%(algo)
)

And inside Testing/CMake/ArtifactoryExternalData.cmake it would look like:

function(ARTIFACTORY_DOWNLOAD_FILE file algo hash)
  set(json_response_file
${CMAKE_BINARY_DIR}/artifactory_search_${algo}_${hash}.json)
  set(artifact_search_query
${ARTIFACTORY_BASE_URL}/api/search/checksum?${algo}=${hash})

  message(Query URI: ${artifact_search_query})

  file(DOWNLOAD ${artifact_search_query} ${json_response_file} STATUS
status)
  list(GET status 0 error_code)
  list(GET status 1 error_message)
 if(error_code)
   file(REMOVE ${json_response_file})
   message(FATAL_ERROR Artifactory query failed with error ${error_code}:
${error_message})
 endif()

  # For testing
  file(WRITE ${json_response_file} {\results\:[{\uri\:\
https://example.com/artifactory/api/storage/repo1-cache/lena.png\}]};)

  file(READ ${json_response_file} json_response)
  file(REMOVE ${json_response_file})

  # Normalize the JSON response by removing any whitespace (makes it easier
to parse)
  string(REGEX REPLACE [ \t\n\r]+  json_response ${json_response})

  if(NOT json_response MATCHES \uri\:)
message(FATAL_ERROR Artifactory file was not found)
  endif()

  string(REGEX MATCH https?://[^\]+ artifact_uri ${json_response})
  message(Artifact URI: ${artifact_uri})

  file(DOWNLOAD ${artifact_uri} ${file} STATUS status)
  list(GET status 0 error_code)
  list(GET status 1 error_message)
  if(error_code)
message(FATAL_ERROR Artifactory download failed with error
${error_code}: ${error_message})
  endif()
endfunction()

Then if you had:

ExternalData_Add_Test(MyProjectData
  NAME SmoothingTest
  COMMAND SmoothingExe DATA{Input/Image.png}
   SmoothedImage.png
  )

The ExternalData framework would execute:

ARTIFACTORY_DOWNLOAD_FILE(${ExternalData_BINARY_ROOT}/Input/Image.png md5
1234567890abcdef1234567890abcdef)

Thoughts?

Taylor
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake-developers

[CMake] Integrating ExternalData with Artifactory

2014-09-19 Thread Taylor Braun-Jones 
Hi all,

I'm curious if anyone has attempted to integrate Artifactory with the
ExternalData framework?

Unfortunately the Artifactory REST API is a two step process where you
first search for a file based on MD5 or SHA1 hash[1] which returns JSON
results the file's URI. You then issue a second request to download the
file from that URI. Any ideas for integrating this into
CMake's ExternalData framework as cleanly as possible?

Thanks!

Taylor

[1]
http://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API#ArtifactoryRESTAPI-ChecksumSearch
-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake

Re: [CMake] Integrating ExternalData with Artifactory

2014-09-19 Thread Brad King 
On 09/19/2014 12:23 PM, Taylor Braun-Jones wrote:
 integrate Artifactory with the ExternalData framework?
 
 the Artifactory REST API is a two step process

Currently ExternalData always uses the file(DOWNLOAD) command.
Recently I was thinking about how to extend the ExternalData
module to support custom download commands or scripts.

I think it can be activated by a special format of an entry in
ExternalData_URL_TEMPLATES that specifies a lookup key that maps
to some kind of custom configuration of a .cmake script to include
or command to launch with execute_process.

If you are interested in design discussion and/or trying to develop
the changes for this, please join the cmake-developers list:

 http://www.cmake.org/mailman/listinfo/cmake-developers

and post this over there.

Thanks,
-Brad

-- 

Powered by www.kitware.com

Please keep messages on-topic and check the CMake FAQ at: 
http://www.cmake.org/Wiki/CMake_FAQ

Kitware offers various services to support the CMake community. For more 
information on each offering, please visit:

CMake Support: http://cmake.org/cmake/help/support.html
CMake Consulting: http://cmake.org/cmake/help/consulting.html
CMake Training Courses: http://cmake.org/cmake/help/training.html

Visit other Kitware open-source projects at 
http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://public.kitware.com/mailman/listinfo/cmake