RE: [VFS] Which Hadoop 2.x .jars are necessary with VFS HDFS access?
Got it working. The list of .jars I needed for Commons VFS to work with Hadoop
2.3 is the following:
commons-cli-1.2.jar
commons-collections-3.2.1.jar
commons-configuration-1.6.jar
commons-lang-2.6.jar
commons-logging-api-1.1.3.jar
commons-vfs2-2.1-SNAPSHOT.jar
guava-11.0.2.jar
hadoop-auth-2.3.0.jar
hadoop-common-2.3.0.jar
hadoop-hdfs-2.3.0.jar
log4j-1.2.17.jar
protobuf-java-2.5.0.jar
slf4j-api-1.7.5.jar
slf4j-log4j12-1.7.5.jar
The other problem was the port number. 50070 is for HTTP access, the port
necessary for this direct HDFS access is 8020.
Thanks to Dave Marion who pointed me in the right direction.
I am now able to browse an HDFS file system from a Windows GUI client (running
an Apache Pivot application) to a Linux Hadoop cluster.
~Roger
-Original Message-
From: Gary Gregory [mailto:garydgreg...@gmail.com]
Sent: Friday, April 11, 2014 2:49 PM
To: Commons Users List
Subject: Re: [VFS] Which Hadoop 2.x .jars are necessary with VFS HDFS access?
FWIW, I've not had luck getting VFS with Hadoop 1.1.2 to work on Windows. I run
the unit tests on a Linux VM. YMMV.
Gary
On Fri, Apr 11, 2014 at 5:07 PM, Roger Whitcomb
wrote:
> My client is on Windows 7, but the servers (Hadoop clusters) are
> mostly Ubuntu (Cloudera VMs) for now.
>
> There doesn't seem to be any problem of Hadoop versions (i.e., it
> finds and loads the Hadoop classes just fine), but it isn't being told
> what the right configuration is.
>
> I've now seem some other postings elsewhere that seem to indicate that
> the FS.defaultFS setting in the Configuration object has to point to
> the remote system. So, I'm guessing that the VFS HdfsProvider needs
> to know some more than I am telling it. Or maybe this is a change from
> Hadoop 1.x to 2.x?!
>
> Thanks,
> ~Roger
>
> -Original Message-
> From: Gary Gregory [mailto:garydgreg...@gmail.com]
> Sent: Friday, April 11, 2014 12:58 PM
> To: Commons Users List
> Subject: Re: [VFS] Which Hadoop 2.x .jars are necessary with VFS HDFS
> access?
>
> Curious: What OS are you on?
>
> We build trunk with Hadoop version 1.1.2, so who know what happens
> with a newer version.
>
> I would start with trying to build VFS with the version of Hadoop you
> want to use to see what happens...
>
> Gary
>
>
> On Fri, Apr 11, 2014 at 3:51 PM, Roger Whitcomb
> wrote:
>
> > Hi Martin,
> > Well, I downloaded Hadoop 2.3.0 from one of the mirrors, and cannot
> > find a "hadoop-corejar" file anywhere in that distribution. But
> > I was able to track down sort of the minimum set of .jars necessary
> > to at least (try to) connect using Commons VFS 2.1:
> > commons-collections-3.2.1.jar
> > commons-configuration-1.6.jar
> > commons-lang-2.6.jar
> > commons-vfs2-2.1-SNAPSHOT.jar
> > guava-11.0.2.jar
> > hadoop-auth-2.3.0.jar
> > hadoop-common-2.3.0.jar
> > log4j-1.2.17.jar
> > slf4j-api-1.7.5.jar
> > slf4j-log4j12-1.7.5.jar
> >
> > What's happening now is that I instantiated the HdfsProvider this way:
> > private static DefaultFileSystemManager manager = null;
> >
> > static
> > {
> > manager = new DefaultFileSystemManager();
> > try {
> > manager.setFilesCache(new DefaultFilesCache());
> > manager.addProvider("hdfs", new HdfsFileProvider());
> > manager.setFileContentInfoFactory(new
> > FileContentInfoFilenameFactory());
> > manager.setFilesCache(new SoftRefFilesCache());
> > manager.setReplicator(new DefaultFileReplicator());
> > manager.setCacheStrategy(CacheStrategy.ON_RESOLVE);
> > manager.init();
> > }
> > catch (final FileSystemException e) {
> > throw new
> > RuntimeException(Intl.getString("object#manager.setupError"), e);
> > }
> > }
> >
> > Then, I try to browse into an HDFS system this way:
> > String url = String.format("hdfs://%1$s:%2$d/%3$s",
> > "hadoop-master ", 50070, hdfsPath);
> > return manager.resolveFile(url);
> >
> > Which results in a bunch of error messages in the log file, which
> > looks like it is trying to do user validation on the local machine
> > instead of against the Hadoop (remote) cluster.
> > Apr 11,2014 18:27:38.640 GMT T[AWT-EventQueue-0](26) DEBUG
> > FileObjectManager: Trying to resolve file reference
> > 'hdfs://hadoop-master:50070/'
> > Apr 11,2014 18:27:38.953 GMT T[AWT-EventQueue-0](26) INFO
> > org.apache.hadoop.conf.Configuration.deprecation: fs.default.name is
> > deprecated. Instead, use fs.defaultFS Apr 11,2014 18:27:39.078 GMT
> > T[AWT-EventQueue-0](26) DEBUG
> > MutableMetricsFactory: field
> > org.apache.hadoop.metrics2.lib.MutableRate
> > org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSucc
> > es
> > s
> > with annotation
> > @org.apache.hadoop.metrics2.annotation.Metric(valueName=Time,
> > value=[Rate of successful kerberos logins and latency
> > (milliseconds
[JXPATH] Am I doing something stupid here?
Hi all,
I'm trying to learn how to use JXPath with DOM in order to speed up some
code that uses a lot of xpath. I've seen blog posts suggesting it's
about twice as fast as JAXP's XPath processor...
The problem I'm running into is when I construct a JXPathContext around
a node down in the DOM tree, then try to select a node elsewhere in the
tree using the ancestor:: axis. I'm attaching a sample XML file and unit
test that shows what I'm trying to do.
I've run this through a debugger, and it appears that the
DOMNodePointer.getImmediateParent() doesn't even try to look at the
Node.getParentNode()...if it doesn't have a pre-existing parent (from
its ctor) then it just dumbly returns the null parent.
I haven't done enough research yet to know how to get DOMNodePointer to
populate its parent (using the public API, not the nuts-and-bolts impl
details), but in the attached example you can see I try two approaches:
1. the naive approach, which is also the last one in the code. IMO, this
one should work!
2. a brute-force alternative, where JXPathContext instances for each
intermediate node are created to inherit in the right order, all the way
back to the document itself. From my partial reading of the code, this
should work even if the naive approach doesn't.
Neither of these works, though. Can someone shed some light on it, or
let me know if I've found a bug (seems like a common use case)...
Thanks,
-john
--
John Casey
GitHub - http://github.com/jdcasey
http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";>
4.0.0
org.test
test-project
1.0
org.group
artifact-id
2.6
package org.commonjava.maven.galley.maven.model.view;
import java.io.InputStream;
import java.io.StringWriter;
import java.util.List;
import java.util.Stack;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.transform.OutputKeys;
import javax.xml.transform.Transformer;
import javax.xml.transform.TransformerFactory;
import javax.xml.transform.dom.DOMSource;
import javax.xml.transform.stream.StreamResult;
import org.apache.commons.jxpath.JXPathContext;
import org.junit.Before;
import org.junit.Test;
import org.w3c.dom.Document;
import org.w3c.dom.Node;
public class JXPathContextAncestryTest
{
@Test
public void basicJXPathTest()
throws Exception
{
final InputStream is = Thread.currentThread()
.getContextClassLoader()
.getResourceAsStream(
"jxpath/simple.pom.xml" );
final Document document = DocumentBuilderFactory.newInstance()
.newDocumentBuilder()
.parse( is );
final JXPathContext ctx = JXPathContext.newContext( document );
document.getDocumentElement()
.removeAttribute( "xmlns" );
final String projectGroupIdPath = "ancestor::project/groupId";
// NOT what's failing...just populating the node set to traverse in
order to feed the ancestor:: axis test.
final List nodes = ctx.selectNodes(
"/project/dependencies/dependency" );
for ( final Object object : nodes )
{
final Node node = (Node) object;
dump( node );
final Stack revPath = new Stack();
Node parent = node;
while ( parent != null )
{
revPath.push( parent );
parent = parent.getParentNode();
}
JXPathContext nodeCtx = null;
while ( !revPath.isEmpty() )
{
final Node part = revPath.pop();
if ( nodeCtx == null )
{
nodeCtx = JXPathContext.newContext( part );
}
else
{
nodeCtx = JXPathContext.newContext( nodeCtx, part );
}
}
System.out.println( "Path derived from context: '" +
nodeCtx.getNamespaceContextPointer()
.asPath() + "'" );
// brute-force approach...try to force population of the parent
pointers by painstakingly constructing contexts for all intermediate nodes.
System.out.println( "Selecting groupId for declaring project using
path-derived context..." );
System.out.println( nodeCtx.getValue( projectGroupIdPath ) );
// Naive approach...this has all the context info it needs to get
parent contexts up to and including the document!
System.out.println( "Selecting groupId for declaring project using
non-derived context..." );
System.out.println( JXPathContext.newC
Re: SCXML retrieve root context
On 12-04-14 21:46, ten...@free.fr wrote:
Thank you for your help Ate, it works very well, I still have a long way to go
in understanding SCXML..
No problem, I'd be happy to help you out further if you have more questions.
If you haven't seen them yet, the slides of my presentation I gave at the
ApacheCon last week might also be helpful [1], especially page 9-13.
I'll update the SCXML website sometime soon to provide a direct link to these
slides as well.
Regards, Ate
[1]
http://events.linuxfoundation.org/sites/events/files/slides/ApacheConUS2014%20-%20Apache%20Commons%20SCXML%202.0.pdf
Francis,
On 12-04-14 10:28, ten...@free.fr wrote:
Hi,
I would like to know if it's possible to retrieve the rootcontext given during
the creation of an SCXMLExecutor inside a custon action (i.e. Hello.class
example).
Yes you can. There isn't a direct API for it but its easy to retrieve the root
context with:
@Override
public void execute(ActionExecutionContext exctx) throws ModelException,
SCXMLExpressionException {
Context context = exctx.getGlobalContext();
while (context.getParent() != null) {
context = context.getParent();
}
// context now references the root context
...
}
The above starts with the global context (the one used and reserved for the
root/script element) and walks the parent tree up. Currently only the system
context sits between the root and global context.
HTH, Ate
-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org
-
To unsubscribe, e-mail: user-unsubscr...@commons.apache.org
For additional commands, e-mail: user-h...@commons.apache.org
