Samiflux commented on PR #8951:
URL: https://github.com/apache/hop/pull/8951#issuecomment-6438064476

   Hi, thanks for this fix! I'm trying to get more involved in Apache Hop 
development whenever I can, although my free time is limited, so apologies for 
the delayed follow-up.
   
   I tested your minimal pipeline (Generate Rows → Generate Random Value, 1000 
rows), and it runs successfully on Windows with Hop 2.20.0-SNAPSHOT, Java 21 
and Spark 4.1.3.
   
   This is a good sign: the Windows fallback allows this minimal pipeline to 
run without setting `HADOOP_HOME` or `hadoop.home.dir`.
   
   However, my pipeline, which I shared earlier in this PR, uses Spark File 
Input → Sample Rows → Spark File Output to read a local CSV file and write the 
result to a local `file:///` path. It still fails with this error:
   
   ```text
   java.lang.UnsatisfiedLinkError: 'boolean 
org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(java.lang.String, int)'
   ```
   
   Full log:
   
   ```text
   2026/10/27 09:58:50 [Basic]    test - Prepared native Spark pipeline engine 
with run configuration 'spark-local' (master=local[*])
   2026/10/27 09:58:50 [Basic]    test - Reusing default SparkSession 
(version=4.1.3)
   2026/10/27 09:58:51 [Error]    test - Error executing Spark pipeline
   2026/10/27 09:58:51 [Error]    test - java.lang.UnsatisfiedLinkError: 
'boolean 
org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(java.lang.String, int)'
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:817)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.hadoop.fs.FileUtil.canRead(FileUtil.java:1415)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.hadoop.fs.FileUtil.list(FileUtil.java:1620)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:802)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:2078)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:2122)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:1020)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:2078)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:2122)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getAllCommittedTaskPaths(FileOutputCommitter.java:334)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJobInternal(FileOutputCommitter.java:404)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:377)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitJob(HadoopMapReduceCommitProtocol.scala:184)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$writeAndCommit$3(FileFormatWriter.scala:275)
   2026/10/27 09:58:51 [Error]    test -        at 
scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.util.Utils$.timeTakenMs(Utils.scala:496)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$.writeAndCommit(FileFormatWriter.scala:275)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeWrite(FileFormatWriter.scala:306)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:189)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:195)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:117)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:115)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:129)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.execution.QueryExecution.$anonfun$eagerlyExecuteCommands$2(QueryExecution.scala:185)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$8(SQLExecution.scala:177)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.execution.SQLExecution$.withSessionTagsApplied(SQLExecution.scala:285)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$7(SQLExecution.scala:139)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.JobArtifactSet$.withActiveJobArtifactState(JobArtifactSet.scala:94)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.artifact.ArtifactManager.$anonfun$withResources$1(ArtifactManager.scala:112)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.artifact.ArtifactManager.withClassLoaderIfNeeded(ArtifactManager.scala:106)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.artifact.ArtifactManager.withResources(ArtifactManager.scala:111)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$6(SQLExecution.scala:139)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:308)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId0$1(SQLExecution.scala:138)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:804)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId0(SQLExecution.scala:92)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:250)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.execution.QueryExecution.$anonfun$eagerlyExecuteCommands$1(QueryExecution.scala:185)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.execution.QueryExecution$.withInternalError(QueryExecution.scala:717)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.execution.QueryExecution.org$apache$spark$sql$execution$QueryExecution$$eagerlyExecute$1(QueryExecution.scala:184)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$3.applyOrElse(QueryExecution.scala:201)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$3.applyOrElse(QueryExecution.scala:194)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:491)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:107)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:491)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:37)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:360)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:356)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:37)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:37)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:467)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:194)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.execution.QueryExecution.$anonfun$lazyCommandExecuted$1(QueryExecution.scala:155)
   2026/10/27 09:58:51 [Error]    test -        at 
scala.util.Try$.apply(Try.scala:217)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.util.Utils$.doTryWithCallerStacktrace(Utils.scala:1392)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.util.LazyTry.tryT$lzycompute(LazyTry.scala:46)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.util.LazyTry.tryT(LazyTry.scala:46)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.util.LazyTry.get(LazyTry.scala:58)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:160)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:239)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.classic.DataFrameWriter.runCommand(DataFrameWriter.scala:592)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.spark.sql.classic.DataFrameWriter.save(DataFrameWriter.scala:115)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.hop.spark.pipeline.handler.SparkFileIoSupport.writeDataset(SparkFileIoSupport.java:119)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.hop.spark.pipeline.handler.SparkFileOutputHandler.handleTransform(SparkFileOutputHandler.java:149)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.hop.spark.pipeline.HopPipelineMetaToSparkConverter.createDataset(HopPipelineMetaToSparkConverter.java:267)
   2026/10/27 09:58:51 [Error]    test -        at 
org.apache.hop.spark.engines.SparkPipelineEngine.lambda$0(SparkPipelineEngine.java:344)
   2026/10/27 09:58:51 [Error]    test -        at 
java.base/java.lang.Thread.run(Thread.java:1583)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to