Re: quick_start.py : no server (connection refused)
Build with -Pberkeley-elasticsearch,dist worked for me. Build with -Pembedded-hbase-solr,dist does NOT work for me. It had issues starting up Solr. Had symtoms similar to ATLAS-800 From: Nigel Jones To: dev@atlas.incubator.apache.org Date: 28/06/16 06:35 PM Subject:quick_start.py : no server (connection refused) There's been some changes to the atlas setup/build in recent months. I just build trunk for the first time in a while, and ran quick_start but the server clearly hasn't started } ] } Exception in thread "main" com.sun.jersey.api.client.ClientHandlerException: java.net.ConnectException: Connection refused at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle (URLConnectionClientHandler.java:155) Following a clean build, I built the distribution with mvn clean package -Pdist,embedded-hbase-solr previously I'd not used the embedded-hbase-solr, but from InstallationSteps.html this now seems the preferred option. I see relatively little in the quick start log: 2016-06-28 12:07:01,129 INFO - [main:] ~ Looking for atlas-application.properties in classpath (ApplicationProperties:73) 2016-06-28 12:07:01,133 INFO - [main:] ~ Loading atlas-application.properties from file:/home/ibmcloud/build/atlas/20160628a/apache-atlas-0.8-incubating-SNAPSHOT/conf/atlas-application.properties (ApplicationProperties:86) 2016-06-28 12:07:14,673 INFO - [main:] ~ Client has only one service URL, will use that for all actions: http://localhost:21000 (AtlasClient:248) 2016-06-28 12:07:15,256 WARN - [main:] ~ Handled exception in calling api api/atlas/types (AtlasClient:725) com.sun.jersey.api.client.ClientHandlerException: java.net.ConnectException: Connection refused at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle (URLConnectionClientHandler.java:155) at com.sun.jersey.api.client.filter.HTTPBasicAuthFilter.handle (HTTPBasicAuthFilter.java:105) Any tips? I'm just going back to the code/scripts to figure out how it's put together, but it may be I've missed some key step? Note - no firewall running or apps on related ports. Centos 7. Thanks Nigel.
Re: Atlas mongoDB integration
To begin with, may be you can start small ... 1) Define the metamodel for MongoDb to capture its data modeling notions like "Document", "Field" etc., (I guess may be one would need a model pretty close to defining a JSON) 2) Once you are done with #1, investigate if there is a way to infer / export the mongodb document structures (schema) in some intermediate format (JSON schema / XSD) 3) Write a translator / interpreter for #2 to convert that into metadata that can be stored in Atlas. Essentially, read JSON schema, persist as instances of types defined in #1 by making Atlas REST calls (may be use AtlasClient) All of the above enables only an offline push of metadata from Mongo to Atlas. For online integration or more real time automated metadata population ... 4) You would need to figure out a way to intercept a hook in MongoDb to write a callback utility when Document(s) are created / updated / deleted / renamed in MongoDb. May be a trigger like mechanism, I guess would certainly be there in MongoDb. (Figure out a hook which is publicly exposed and is widely used) 5) I guess after #4, you can reuse work done in steps #1 to #3. Atlas committers can add more light on integrating closely with Atlas through AtlasHook. Thanks Venkat From: "Ziliotto, Federico" To: "dev@atlas.incubator.apache.org" Date: 10/05/16 05:18 PM Subject:Atlas mongoDB integration Hello everyone, We would be really interested in using Atlas as a metadata storage, to store additional information about the datasets we are using (mostly to discover what kind of data is present, but also to have data quality, authorization information ecc...). Since most of our data is stored in mongodb, I wanted to create a mongoBridge similar to the Hive/Sqoop/Storm available in the Atlas codebase. Do you think it is feasible/useful to do so? What would be the implementation challenges of it or the integration problems with the current environment (for instance Ranger would not be able to enforce any security policy related to it without some custom plugin for both ranger and mongo)? Which would be to proper way to implement such a component: as a custom Atlas installation or as a separate project that uses Atlas? I'm looking for any kind of suggestion or opinion on the topic. Thanks in advance, Federico ** The information in this e-mail (and any attachments) is intended exclusively for the addressee(s). Any use by a party other than the addressee(s) is prohibited. The information may be confidential in nature and fall under a duty of non-disclosure. If you are not the addressee, please notify the sender and delete this e-mail. KPMG cannot guarantee that e-mail communications are secure and error-free and does not accept any liability for damages resulting from the use of e-mail. Our services and other work are carried out under an agreement of instruction (overeenkomst van opdracht) that is subject to the general terms and conditions of the contracting KPMG firm. These general terms and conditions are available on our website (www.kpmg.nl/algemenevoorwaarden) and will be forwarded upon request. The statutory names of the KPMG-firms and the trade register numbers are also stated on our website, on the same page as the general terms. Agreements with and statements from KPMG are only legally binding upon KPMG if they are confirmed in writing and signed by an authorized person.
Re: Questions about REST API and payloads...
Hi Ernie,
Sometime last year, I had put up a small doc trying to understand the
Lineage support in Atlas.
See if this helps.
Thanks
Venkat
(See attached file: Atlas_Lineage_Support.pdf)
From: Hemanth Yamijala
To: "dev@atlas.incubator.apache.org"
Date: 09/05/16 08:30 PM
Subject:Re: Questions about REST API and payloads...
The APIs in question are
atlas/api/lineage/hive/table/{tableName}/inputs/graph and
atlas/api/lineage/hive/table/{tableName}/outputs/graph. The implementation
for the same is HiveLineageResource.java. (The ‘Hive’ in the names is an
unfortunate legacy – it applies to any type in Atlas that extends from a
DataSet). The UI code that shows the lineage graphs (in the older dashboard
module) should probably help see how the outputs of the two APIs can be
used for getting a lineage graph in place.
Thanks
Hemanth
From: Ernie Ostic mailto:eos...@us.ibm.com>>
Reply-To: "dev@atlas.incubator.apache.org<
mailto:dev@atlas.incubator.apache.org>" mailto:dev@atlas.incubator.apache.org>>
Date: Monday, May 9, 2016 at 8:19 PM
To: "dev@atlas.incubator.apache.org"
mailto:dev@atlas.incubator.apache.org>>
Subject: Re: Questions about REST API and payloads...
Hi...
New to the list, so may have missed a response to the original note below,
but in the meantime, also had this question relative to the REST API. It
doesn't appear that there is currently a published API for declaring
lineage? (for connecting assets together). Is it just not yet documented?
Are there plans for exposing lineage definition via REST?
Thank you.
Ernie
Ernie Ostic
WW Product Specialist, Information Server
IBM Analytics
Cell: (617) 331 8238
---
Open IGC is here!
Extend the Catalog with custom objects and lineage definitions!
https://dsrealtime.wordpress.com/2015/07/29/open-igc-is-here/
[Inactive hide details for Ernie Ostic---05/06/2016 01:39:04 PM---Hello...
Thanks to everyone for all the useful information on]Ernie
Ostic---05/06/2016 01:39:04 PM---Hello... Thanks to everyone for all the
useful information on the Apache Atlas site to-date. Recen
From: Ernie Ostic/Newark/IBM
To: dev@atlas.incubator.apache.org
Date: 05/06/2016 01:39 PM
Subject: Question about REST payloads...
Hello...
Thanks to everyone for all the useful information on the Apache Atlas site
to-date. Recently I started experimenting with Atlas, using the REST API,
and establishing some of my own types and then reviewing their metadata. I
was able to get things working for the currently published REST calls, but
would like to know more about the formatting of the payloads. The returned
JSON payload is valid but embeds further JSON that "appears" to have
additional formatting --- is this something that is handled automatically
by some of the (presumably java) existing client tooling? ...or maybe I
should have passed another argument or made a different call that would
pass back a straight text response?
Here is a snippet of the sample response I received
{"requestId":"qtp430472391-17 -
12f1518c-fdac-4a50-a23a-f56af47ee069","GUID":"1731e6e8-ee11-4fc0-b86a-e83f08d67981","definition":"{\n
\"jsonClass\":\"org.apache.atlas.typesystem.json.InstanceSerialization
$_Reference\",\n \"id\":{\n \"jsonClass
\":\"org.apache.atlas.typesystem.json.InstanceSerialization$_Id\",\n \"id
\":\"1731e6e8-ee11-4fc0-b86a-e83f08d67981\"
Is there a historical reason for the further formatting of "definition"? Is
there a parameter I should be passing that I accidentally missed (that
would avoid this formatting)?
Thanks. Many of our customers are considering using REST calls from a
variety of tools and languages. This structure will require additional
parsing and formatting after breaking out the initial elements.
Ernie
Ernie Ostic
WW Product Specialist, Information Server
IBM Analytics
Cell: (617) 331 8238
---
Open IGC is here!
Extend the Catalog with custom objects and lineage definitions!
https://dsrealtime.wordpress.com/2015/07/29/open-igc-is-here/
Re: FYI - What is FieldMapping class in TypeSystem
Sorry - typing in Notepad added undesired TABs, hope this would one looks
good.
While working on ATLAS-683, I had to understood the need of the class named
'FieldMapping' in the typesystem
project. Owing to the lack of javadoc, I had to trace its usage points to
figure out the need for it.
Incase anyone wants to know, here is my understanding... (Will try and
explain with simple code. Note
that Attribute/Field words are used interchangeably)
Lets just restrict to types which can only hold attributes of primitive
types (no references). In that
case, the ClassType would like..
class ClassType {
List attributes;
}
class Attribute {
String name;
String dataType; // primitive
}
An instance of ClassType can be described like...
class Instance {
Map values; // 'key' is attribute name and 'value' is
the
// primitive value of it like Boolean, Integer, Long,
String etc.,
Object getValue(String attrName) {
return values.get(attrName);
}
}
Instead of storing all values in a single Map, one can partition the values
in buckets based on the 'type' of the value as follows...
class Instance {
Boolean[] booleanValues;
String[] stringValues;
Integer intValues;
...
}
If one has to persist the values in the above partitioned buckets, how does
one query the value
of an attribute. One would need to know the 'type' of the attribute as well
as the 'position'
of the attribute.
class Instance {
Boolean[] booleanValues;
String[] stringValues;
..
Object getBooleanValue(String attrName) {
int valuePosition = ... // get it from somewhere, where ? see below
return booleanValues[valuePosition];
}
}
In the above code, the position of the boolean attribute can be captured in
a class like
FieldMapping as follows...
class FieldMapping {
Map fieldPos; // 'key' is attribute name and 'value'
is the position of
// the 'attribute'.
}
In the Instance class above, the array 'booleanValues' needs to be
initialized with a fixed size
array with the total count equal to the number of 'boolean' valued
attributes in ClassType.
So we extend the FieldMapping a little further to precompute the total
number of boolean values.
class FieldMapping {
Map fieldPos;
int numBools; // boolean attributes count
int numStrings; // string attributes count
...
}
From: Venkata R Madugundu/India/IBM@IBMIN
To: atlas
Date: 25/04/16 12:15 PM
Subject:FYI - What is FieldMapping class in TypeSystem
While working on ATLAS-683, I had to understood the need of the class named
'FieldMapping' in the typesystem
project. Owing to the lack of javadoc, I had to trace its usage points to
figure out the need for it.
Incase anyone wants to know, here is my understanding... (Will try and
explain with simple code. Note
that Attribute/Field words are used interchangeably)
Lets just restrict to types which can only hold attributes of primitive
types (no references). In that
case, the ClassType would like..
class ClassType {
List attributes;
}
class Attribute {
String name;
String dataType; // primitive
}
An instance of ClassType can be described like...
class Instance {
Map values; // 'key' is attribute name and
'value' is
the
//
primitive value of
it like Boolean, Integer,
Long, String etc.,
Object getValue(String attrName) {
return values.get(attrName);
}
}
Instead of storing all values in a single Map, one can partition the values
in buckets based on the 'type' of the value as follows...
class Instance {
Boolean[] booleanValues;
String[] stringValues;
Integer intValues;
...
}
If one has to persist the values in the above partitioned buckets, how does
one query the value
of an attribute. One would need to know the 'type' of the attribute as well
as the 'position'
of the attribute.
class Instance {
Boolean[] booleanValues;
String[] stringValues;
..
Object getBooleanValue(String attrName) {
int valuePosition = ... // get it from
somewhere,
where ? see
below
return booleanValues[valuePosition];
}
}
In the above code, the position of the boolean attribute can be captured in
a class like
FieldMapping as follows...
class FieldMapping {
Map fieldPos; // 'key' is attribute name and
'value'
is the position of
FYI - What is FieldMapping class in TypeSystem
While working on ATLAS-683, I had to understood the need of the class named
'FieldMapping' in the typesystem
project. Owing to the lack of javadoc, I had to trace its usage points to
figure out the need for it.
Incase anyone wants to know, here is my understanding... (Will try and
explain with simple code. Note
that Attribute/Field words are used interchangeably)
Lets just restrict to types which can only hold attributes of primitive
types (no references). In that
case, the ClassType would like..
class ClassType {
List attributes;
}
class Attribute {
String name;
String dataType; // primitive
}
An instance of ClassType can be described like...
class Instance {
Map values; // 'key' is attribute name and 'value' is
the
// primitive value of it like Boolean,
Integer,
Long, String etc.,
Object getValue(String attrName) {
return values.get(attrName);
}
}
Instead of storing all values in a single Map, one can partition the values
in buckets based on the 'type' of the value as follows...
class Instance {
Boolean[] booleanValues;
String[] stringValues;
Integer intValues;
...
}
If one has to persist the values in the above partitioned buckets, how does
one query the value
of an attribute. One would need to know the 'type' of the attribute as well
as the 'position'
of the attribute.
class Instance {
Boolean[] booleanValues;
String[] stringValues;
..
Object getBooleanValue(String attrName) {
int valuePosition = ... // get it from somewhere, where ? see
below
return booleanValues[valuePosition];
}
}
In the above code, the position of the boolean attribute can be captured in
a class like
FieldMapping as follows...
class FieldMapping {
Map fieldPos; // 'key' is attribute name and 'value'
is the position of
// the 'attribute'.
}
In the Instance class above, the array 'booleanValues' needs to be
initialized with a fixed size
array with the total count equal to the number of 'boolean' valued
attributes in ClassType.
So we extend the FieldMapping a little further to precompute the total
number of boolean values.
class FieldMapping {
Map fieldPos;
int numBools; // boolean attributes count
int numStrings; // string attributes count
...
}
Finally, when one thinks of the intent to store in buckets, it is obvious
that it is
for performance reasons. Performance would probably come into picture, when
there are too many attributes in
a given ClassType and the value lookup on the instance needs to be fast.
But the downside of that is, each Instance
holds fixed size buckets based on Type information, which is "constant". So
even if a single value is set in an instance,
the Instance is pre-sized large enough to hold all the values. I am
thinking the current design is this way. When one
looks at the case of a 'query' where a subset of columns/fields are
queried, one would only need to populate the
Instance with those values (which is probably the most frequent usecase).
Thanks
Venkat
Re: [ANNOUNCE] New Committer: Hemanth Yamijala
Congrats Hemanth. From: Seetharam Venkatesh To: dev@atlas.incubator.apache.org Date: 14/04/16 11:26 AM Subject:Re: [ANNOUNCE] New Committer: Hemanth Yamijala Thanks Shwetha for doing the honors and Congratulations Hemanth! On Wed, Apr 13, 2016 at 10:15 PM Shwetha Shivalingamurthy < sshivalingamur...@hortonworks.com> wrote: > Hi All, > > The Apache Atlas PPMC has asked Hemanth Yamijala to become a committer and > we are pleased to announce that he has accepted. > > Regards, > Shwetha > >
Re: [PROPOSAL] Java/Scala API on top of the REST API ?
Here is the Swagger JIRA - https://issues.apache.org/jira/browse/ATLAS-328 By the way, per my last observation, the Atlas REST documentation is a bit dated. To keep it really upto date, generating a Swagger driven documentation which builds off the Java REST API definition will always keep it in sync. For now, to know the exact REST end point, I have only been consulting the Atlas code (java REST resources like TypesResource.java, EntityResource.java, MetadataDiscoveryResource.java, HiveLineageResource.java etc.,) To see what all clients Swagger can generate, go to http://editor.swagger.io/#/ and click on the menu "Generate Client" on the very top. It shows you 30 odd languages in which a client can be generated, and Java/Scala/Python/Javascript are few among them. Thanks Venkat From: Jean-Baptiste Onofré To: dev@atlas.incubator.apache.org Date: 31/03/16 08:39 PM Subject:Re: [PROPOSAL] Java/Scala API on top of the REST API ? Hi Venkata, tanks, I missed this when taking a look on the code. I will test the integration then, and I will work on the Swagger integration. I don't find a Jira about that, should I create one ? Thanks ! Regards JB On 03/31/2016 04:59 PM, Venkata R Madugundu wrote: > There is an AtlasClient.java which wraps around the REST API if you want > to take a look at. > \client\src\main\java\org\apache\atlas\AtlasClient.java > > Also take a look at > \webapp\src\main\java\org\apache\atlas\examples\QuickStart.java to see > how AtlasClient can be used. > > There is a JIRA open for Swagger integration so that one can generate > typed API bindings to interact with Atlas REST API. > As Atlas REST is JAX RS compliant, one can actually make Swagger > generate bindings for different languages (atleast we did > it for Javascript and Python) > > For java based REST calls, we have used google gson for JSON > serialization/deserialization and Apache HttpClient to dispatch REST > requests. > > > Inactive hide details for Jean-Baptiste Onofré ---31/03/2016 07:44:31 > PM---Hi guys, I like the Atlas REST API, and the way it'sJean-Baptiste > Onofré ---31/03/2016 07:44:31 PM---Hi guys, I like the Atlas REST API, > and the way it's documented. > > From: Jean-Baptiste Onofré > To: dev@atlas.incubator.apache.org > Date: 31/03/16 07:44 PM > Subject: [PROPOSAL] Java/Scala API on top of the REST API ? > > > > > > Hi guys, > > I like the Atlas REST API, and the way it's documented. > > In some case, to simplify the integration of Atlas in third party > applications, maybe it would be great to provide a Java and Scala APIs > on top of the REST API (wrapping the transport, and adding some > "control" and DataObject/JSON). > For instance, I'm PoCing some shell & Karaf commands to manipulate Atlas > (as we did in Falcon for instance). In that case, even if REST API is > convenient, it's not as straight forward as a turnkey API. > For instance, it's what we did in Syncope: we provide syncope-client on > top of the Syncope REST API. > > Thoughts ? > > Regards > JB > -- > Jean-Baptiste Onofré > jbono...@apache.org > http://blog.nanthrax.net > Talend - http://www.talend.com > > > -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com
Re: Atlas build error - npm install --color=false failed
Not sure if any others have encountered the same problem. But atleast two members in our team did encounter this. Give me some time. I will update the InstallationSteps.twiki with an additional note. Thanks Venkat From: Shwetha Shivalingamurthy To: "dev@atlas.incubator.apache.org" Date: 01/04/16 09:54 AM Subject:Re: Atlas build error - npm install --color=false failed We should document this in InstallationSteps.twiki. Can you update? Regards, Shwetha On 31/03/16, 3:49 PM, "Venkata R Madugundu" wrote: > >FYI > >With latest sources of Atlas, I am seeing the following build error (on >Windows7) ... > >--- > > > > [ERROR] Failed to execute goal >com.github.eirslett:frontend-maven-plugin:0.0.23:npm (npm install) on >project atlas-dashboard: Failed to run task: > 'npm install --color=false' failed. (error code 50) -> [Help 1] >--- > > > > > After a bit of searching around on web, someone hinted to run cmd as >'Run as Administrator' to resolve 'npm' related maven build errors. > > And that helped incase anyone comes across a similar error. > > > > > >
Re: [PROPOSAL] Java/Scala API on top of the REST API ?
There is an AtlasClient.java which wraps around the REST API if you want to take a look at. \client\src\main\java\org\apache\atlas\AtlasClient.java Also take a look at \webapp\src\main\java\org\apache\atlas\examples \QuickStart.java to see how AtlasClient can be used. There is a JIRA open for Swagger integration so that one can generate typed API bindings to interact with Atlas REST API. As Atlas REST is JAX RS compliant, one can actually make Swagger generate bindings for different languages (atleast we did it for Javascript and Python) For java based REST calls, we have used google gson for JSON serialization/deserialization and Apache HttpClient to dispatch REST requests. From: Jean-Baptiste Onofré To: dev@atlas.incubator.apache.org Date: 31/03/16 07:44 PM Subject:[PROPOSAL] Java/Scala API on top of the REST API ? Hi guys, I like the Atlas REST API, and the way it's documented. In some case, to simplify the integration of Atlas in third party applications, maybe it would be great to provide a Java and Scala APIs on top of the REST API (wrapping the transport, and adding some "control" and DataObject/JSON). For instance, I'm PoCing some shell & Karaf commands to manipulate Atlas (as we did in Falcon for instance). In that case, even if REST API is convenient, it's not as straight forward as a turnkey API. For instance, it's what we did in Syncope: we provide syncope-client on top of the Syncope REST API. Thoughts ? Regards JB -- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com
Atlas build error - npm install --color=false failed
FYI With latest sources of Atlas, I am seeing the following build error (on Windows7) ... --- [ERROR] Failed to execute goal com.github.eirslett:frontend-maven-plugin:0.0.23:npm (npm install) on project atlas-dashboard: Failed to run task: 'npm install --color=false' failed. (error code 50) -> [Help 1] --- After a bit of searching around on web, someone hinted to run cmd as 'Run as Administrator' to resolve 'npm' related maven build errors. And that helped incase anyone comes across a similar error.
Re: Atlas dev environment setup
Thanks Neeru. Here is what I did. (May be it is useful to others) 1. Pulled atlas source from Git. 2. Installed Scala IDE 4.3.0 from http://scala-ide.org/download/sdk.html 3. Imported atlas projects as Maven projects in Eclipse 4. Added the scala folders in 'typesystem' and 'repository' projects as 'Source folders' Right click the scala folder -> Build Path -> Use as Source Folder 5. Changed the Scala Compiler Settings for 'typesystem' and 'repository' projects to use Scala 2.10 Right click project -> Scala Compiler -> Use Project Settings ... select 'Fixed Scala installation 2.10.5' 6. Pretty much every project is green except for 'atlas-webapp' where there is an error flagged in pom. "Plugin execution not covered by lifecycle configuration: org.codehaus.mojo:keytool-maven-plugin:1.5:clean (execution: clean, phase: generate-resources) pom.xml" I think that keytool plugin probably needs some kind of integration with m2eclipse. I am OK to live with this red flag. NOTE: - I have excluded projects in atlas/addons to begin with. - Used Java8 and Maven 3.3.3 Thanks Venkat From: "Fnu Neerju" To: dev@atlas.incubator.apache.org Date: 22/02/16 09:06 PM Subject:Re: Atlas dev environment setup Hi Venkat, I had the same issue. I had to select scala folders and add those on classpath manually. I did it for typesystem and repository projects for atlas. Please choose scala for both main code and test code. PFA the SS. -Neeru(See attached file: ScalainPath.png) Inactive hide details for "Venkata R Madugundu" ---02/22/2016 10:01:22 AM---Is there an Atlas page for the people who want to c"Venkata R Madugundu" ---02/22/2016 10:01:22 AM---Is there an Atlas page for the people who want to contribute, to setup a development environment wit From: "Venkata R Madugundu" To: dev@atlas.incubator.apache.org Date: 02/22/2016 10:01 AM Subject: Atlas dev environment setup Is there an Atlas page for the people who want to contribute, to setup a development environment with Eclipse as IDE ? I am trying to set it up, but one specific issue which keeps me bothering is "the Scala sources do not get recognized inside Eclipse as source folders. I searched a bit on the web and found out that the scala-maven-plugin needs to be configured specifically to let maven eclipse plugin add the Scala source folders as Eclipse source folders. Something like described in http://stackoverflow.com/questions/23794398/eclipse-doesnt-recognize-scala-source-directory But I would prefer to get some advice from already contributing folks on what is your routine procedure to setup Atlas dev env with Eclipse IDE. Thanks Venkat
Atlas dev environment setup
Is there an Atlas page for the people who want to contribute, to setup a development environment with Eclipse as IDE ? I am trying to set it up, but one specific issue which keeps me bothering is "the Scala sources do not get recognized inside Eclipse as source folders. I searched a bit on the web and found out that the scala-maven-plugin needs to be configured specifically to let maven eclipse plugin add the Scala source folders as Eclipse source folders. Something like described in http://stackoverflow.com/questions/23794398/eclipse-doesnt-recognize-scala-source-directory But I would prefer to get some advice from already contributing folks on what is your routine procedure to setup Atlas dev env with Eclipse IDE. Thanks Venkat
Re: Atlas binaries
Thanks Erik, Hemanth for your response. It is essentially the distribution we are looking for. I did a scan of Atlas root pom. The section below seems to give me what I am looking for. default https://repository.apache.org/content/groups/public/ https://repository.apache.org/content/groups/public/org/apache/atlas/atlas-distro/ Thanks Venkat From: Erik Bergenholtz To: "dev@atlas.incubator.apache.org" Date: 18/02/16 05:54 AM Subject:Re: Atlas binaries Venkat, Nightly snapshot builds of Atlas [and other projects] are available in nexus: https://repository.apache.org/#nexus-search;quick~org.apache.atlas. Find what you are looking for and you can download the jars. There is no distro per-se. Erik On Feb 17, 2016, at 1:38 PM, Hemanth Yamijala mailto:hyamij...@hortonworks.com>> wrote: Venkat, If you mean the entire distro - they are available here: https://builds.apache.org/job/apache-atlas-nightly/lastBuild/artifact/distro/target/ - Can you see if this works? However, if you mean a maven repo to which we publish latest SNAPSHOT jars, I am unaware of any - maybe someone else can help? Thanks hemanth On 2/17/16, 2:17 AM, "Venkata R Madugundu" mailto:venkataraman...@in.ibm.com>> wrote: Hi, If I would like to get the Atlas binaries from the current source snapshot (or a release), is there a publicly accessible Maven repository from where I can download. I know there are instructions to build from source on the Atlas page, but would like to know if the nightly build deploys the artifacts to any publicly accessible Maven repository. Thanks Venkata Madugundu
Atlas binaries
Hi, If I would like to get the Atlas binaries from the current source snapshot (or a release), is there a publicly accessible Maven repository from where I can download. I know there are instructions to build from source on the Atlas page, but would like to know if the nightly build deploys the artifacts to any publicly accessible Maven repository. Thanks Venkata Madugundu
Re: Atlas service scalability
Hi Hemanth, Thanks very much for that quick reply. > Are you thinking of horizontal scale for the Atlas server because > of the load of Metadata queries from clients? Could you share your > use case in some more detail? You are right. We are expecting the quantum of metadata persisted/accessed by clients to be very large (and growing). And hence a requirement for horizontal distribution of Atlas runtime. The concrete usecases we are looking at... a) Cloud deployment for a multi-tenant metadata repository. A single metadata repository would be shared by multiple tenants but with metadata isolation per tenant. In other words, the metadata repository is more or less seen as a regular database (rather an 'Object' database) for analytics applications on cloud. b) On-prem deployment for large enterprises which are looking for consolidation of enterprise wide metadata into a single metadata repository. The metadata aggregation is achieved with periodic (daily) synchronization of metadata using long running metadata imports. > Given the shared state implementation we have, there are no > workarounds immediately, I am afraid. OK. I am assuming, there is some sort of optimization (likely caching) done in the Atlas code and hence that statefullness of Atlas which does not allow multiple active instances. Thanks Venkata Madugundu Hemanth Yamijala wrote on 25/01/2016 05:18:26 PM: > From: Hemanth Yamijala > To: "dev@atlas.incubator.apache.org" > Date: 25/01/16 05:18 PM > Subject: Re: Atlas service scalability > > Venkata, > > Answers inline. > > Thanks > Hemanth > > > > > > On 1/25/16, 4:56 PM, "Venkata R Madugundu" wrote: > > > > > > >Hi, > > > >We are evaulating the ways in which Atlas can scale. I am looking for way > >in which I can run multiple instances of Atlas service/runtime for > >horizontal scale (An instance running per node in a cluster). Is this > >possible currently ? > > As you note below, the Atlas Service is not currently horizontally > scalable. However, the backends which store the metadata (HBase, > Solr etc) can be configured to achieve horizontal scale. > > > > >I went through the Atlas page on 'Fault tolerance and HA'. It clearly says > >"Currently, the Atlas Web service has a limitation that it can only have > >one active instance at a time". I also went through one post on the > >mailing list related to this with the subject 'Atlas with hbase/solr/kafka' > >which seems to say there can be only one active instance of Atlas service > >at any given point of time with the current state of Atlas implementation. > > > >So, can you please confirm that my understanding is correct, that for now I > >can only run one active Atlas instance at any given point of time (in other > >words multiple instances of Atlas service cannot be running simultaneously > >in a cluster to achieve horizontal scale). If this is correct, are there > >any interim workarounds to the solution that one can make use of ? > > Given the shared state implementation we have, there are no > workarounds immediately, I am afraid. Are you thinking of horizontal > scale for the Atlas server because of the load of Metadata queries > from clients? Could you share your use case in some more detail? > > > > >Thanks > >Venkata Madugundu
Atlas service scalability
Hi, We are evaulating the ways in which Atlas can scale. I am looking for way in which I can run multiple instances of Atlas service/runtime for horizontal scale (An instance running per node in a cluster). Is this possible currently ? I went through the Atlas page on 'Fault tolerance and HA'. It clearly says "Currently, the Atlas Web service has a limitation that it can only have one active instance at a time". I also went through one post on the mailing list related to this with the subject 'Atlas with hbase/solr/kafka' which seems to say there can be only one active instance of Atlas service at any given point of time with the current state of Atlas implementation. So, can you please confirm that my understanding is correct, that for now I can only run one active Atlas instance at any given point of time (in other words multiple instances of Atlas service cannot be running simultaneously in a cluster to achieve horizontal scale). If this is correct, are there any interim workarounds to the solution that one can make use of ? Thanks Venkata Madugundu
