[ https://issues.apache.org/jira/browse/YARN-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16867197#comment-16867197 ]
Akira Ajisaka commented on YARN-9016: ------------------------------------- Hi [~Sushil-K-S] and [~vrushalic], org.json:json is included via azure-documentdb:1.16.2 and it is JSON licensed. The JSON License is now in category X, so would you please remove the json library? https://apache.org/legal/resolved.html#category-x I filed HADOOP-16381 for tracking this issue. > DocumentStore as a backend for ATSv2 > ------------------------------------ > > Key: YARN-9016 > URL: https://issues.apache.org/jira/browse/YARN-9016 > Project: Hadoop YARN > Issue Type: New Feature > Components: ATSv2 > Reporter: Sushil Ks > Assignee: Sushil Ks > Priority: Major > Fix For: 3.3.0 > > Attachments: YARN-9016.001.patch, YARN-9016.002.patch, > YARN-9016.003.patch, YARN-9016.004.patch > > > h1. Document Store for ATSv2 > The Document Store for ATSv2 is a framework for plugging in > any Document Store Vendor as a backend for ATSv2 i.e Azure CosmosDB , > MongoDB, ElasticSearch etc. > * Supports multiple Document Store Vendors like CosmosDB, ElasticSearch, > MongoDB etc by just adding new configurations properties and writing Document > Store reader and writer clients. > * Currently has support for CosmosDB. > * All writes are Async and buffered, latest document would be flushed to the > store either if the document buffer gets full or periodically at every flush > interval in background without adding any additional latency to the running > jobs.. > * All the REST API's of Timeline Reader Server are supported. > h3. *How to enable?* > Add the flowing properties under *yarn-site.xml* > {code:java} > <!-- config required for ATSv2 to use DocumentStore--> > <property> > <name>yarn.timeline-service.writer.class </name> > > <value>org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineWriterImpl</value> > </property> > <property> > <name>yarn.timeline-service.reader.class </name> > <value>org.apache.hadoop.yarn.server.timelineservice.storage.documentstore.DocumentStoreTimelineReaderImpl</value> > </property> > <property> > <name>yarn.timeline-service.document-store.db-name</name> > <value>YOUR_DATABASE_NAME</value> <!-- default is timeline_service --> > </property>{code} > h3. *Creating DB and Collections for storing documents* > The following config needs to be set inside > *yarn-site.xml* for creating the database and collections for storing > documents. > {code:java} > <!-- Using schema creator class for DocumentStore--> > <property> > <name>yarn.timeline-service.schema-creator.class </name> > > <value>org.apache.hadoop.yarn.server.timelineservice.documentstore.DocumentStoreCollectionCreator</value> > </property>{code} > Running the schema creator tool to create the necessary > collections. > {code:java} > bin/hadoop > org.apache.hadoop.yarn.server.timelineservice.storage.TimelineSchemaCreator{code} > h3. *Azure CosmosDB* > To use Azure CosmosDB as a DocumentStore for ATSv2, the additional > properties under *yarn-site.xml* is required.. > {code:java} > <!-- config required for using Azure CosmosDB as a DocumentStore for ATSv2 --> > <property> > <name>yarn.timeline-service.document-store-type</name> > <value>COSMOS_DB</value> > </property> > <property> > <name>yarn.timeline-service.document-store.cosmos-db.endpoint</name> > <value>http://YOUR_AZURE_COSMOS_DB_URL:443/</value> > </property> > <property> > <name>yarn.timeline-service.document-store.cosmos-db.masterkey</name> > <value>YOUR_AZURE_COSMOS_DB_MASTER_KEY_CREDENTIAL</value> > </property> > {code} > > *Testing locally* > In order to test the Azure CosmosDB as a DocumentStore > locally, install the emulator from > [here|https://docs.microsoft.com/en-us/azure/cosmos-db/local-emulator] and > start it locally. Set the endpoint and master key under *yarn-site.xml* as > mentioned above and run any example job like DistributedShell etc. Later you > can check the data explorer UI of Azure CosmosDB locally to query the > documents or even launch the *TimelineReader* locally to fetch/query the data > from REST API's. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org