[jira] [Created] (NIFI-12624) ReportingTask: AWSCloudWatchReporterTask

[Jorge Machado (Jira)]

Jorge Machado created NIFI-12624:


 Summary: ReportingTask: AWSCloudWatchReporterTask
 Key: NIFI-12624
 URL: https://issues.apache.org/jira/browse/NIFI-12624
 Project: Apache NiFi
  Issue Type: Wish
  Components: Core Framework
Affects Versions: 1.24.0
Reporter: Jorge Machado


Hey everyone, we already have a PrometheusReportingTask and an 
AzureLogAnalyticsReportingTask, it would be great if we could push all the 
metrics from PrometheusReportingTask into AWS Cloudwatch. What do you think ? 
Currently there is not straight up way of doing this. If yes where should we 
put it ?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (SPARK-44108) Cannot parse Type from german "umlaut"

[Jorge Machado (Jira)]

 [ 
https://issues.apache.org/jira/browse/SPARK-44108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jorge Machado updated SPARK-44108:
--
Description: 
Hello all, 

 

I have a client that has a column named : bfzgtäeil

Spark cannot handle this. My test: 

 
{code:java}
import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
import org.scalatest.funsuite.AnyFunSuite

class HiveTest extends AnyFunSuite {

  test("test that Spark does not cut columns with ä") {
val data =
  "bfzugtäeil:string"
CatalystSqlParser.parseDataType(data)
  }

} {code}
I debugged it and I'm deep on the  org.antlr.v4.runtime.Lexer class. 

Any ideas ? 

 
{code:java}
== SQL ==bfzugtäeil:string--^^^
at 
org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:306)
   at 
org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:143)
  at 
org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parseDataType(ParseDriver.scala:41)
   at 
com.deutschebahn.zod.fvdl.commons.spark.app.captured.HiveTest2.$anonfun$new$1(HiveTest2.scala:13)
 {code}

  was:
Hello all, 

 

I have a client that has a column named : bfzgtäeil

Spark cannot handle this. My test: 

 
{code:java}
import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
import org.scalatest.funsuite.AnyFunSuite

class HiveTest extends AnyFunSuite {

  test("test that Spark does not cut columns with ä") {
val data =
  "bfzugtäeil:string"
CatalystSqlParser.parseDataType(data)
  }

} {code}
I debugged it and I'm deep on the  org.antlr.v4.runtime.Lexer class. 

Any ideas ? 


> Cannot parse Type from german "umlaut"
> --
>
> Key: SPARK-44108
> URL: https://issues.apache.org/jira/browse/SPARK-44108
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.3.0
>Reporter: Jorge Machado
>Priority: Major
>
> Hello all, 
>  
> I have a client that has a column named : bfzgtäeil
> Spark cannot handle this. My test: 
>  
> {code:java}
> import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
> import org.scalatest.funsuite.AnyFunSuite
> class HiveTest extends AnyFunSuite {
>   test("test that Spark does not cut columns with ä") {
> val data =
>   "bfzugtäeil:string"
> CatalystSqlParser.parseDataType(data)
>   }
> } {code}
> I debugged it and I'm deep on the  org.antlr.v4.runtime.Lexer class. 
> Any ideas ? 
>  
> {code:java}
> == SQL ==bfzugtäeil:string--^^^
>   at 
> org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:306)
>at 
> org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:143)
>   at 
> org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parseDataType(ParseDriver.scala:41)
>at 
> com.deutschebahn.zod.fvdl.commons.spark.app.captured.HiveTest2.$anonfun$new$1(HiveTest2.scala:13)
>  {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-44108) Cannot parse Type from german "umlaut"

[Jorge Machado (Jira)]

 [ 
https://issues.apache.org/jira/browse/SPARK-44108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jorge Machado updated SPARK-44108:
--
Description: 
Hello all, 

 

I have a client that has a column named : bfzgtäeil

Spark cannot handle this. My test: 

 
{code:java}
import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
import org.scalatest.funsuite.AnyFunSuite

class HiveTest extends AnyFunSuite {

  test("test that Spark does not cut columns with ä") {
val data = "bfzugtäeil:string"
CatalystSqlParser.parseDataType(data)
  }

} {code}
I debugged it and I'm deep on the  org.antlr.v4.runtime.Lexer class. 

Any ideas ? 

 
{code:java}
== SQL ==bfzugtäeil:string--^^^
at 
org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:306)
   at 
org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:143)
  at 
org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parseDataType(ParseDriver.scala:41)
   at 
com.deutschebahn.zod.fvdl.commons.spark.app.captured.HiveTest2.$anonfun$new$1(HiveTest2.scala:13)
 {code}

  was:
Hello all, 

 

I have a client that has a column named : bfzgtäeil

Spark cannot handle this. My test: 

 
{code:java}
import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
import org.scalatest.funsuite.AnyFunSuite

class HiveTest extends AnyFunSuite {

  test("test that Spark does not cut columns with ä") {
val data =
  "bfzugtäeil:string"
CatalystSqlParser.parseDataType(data)
  }

} {code}
I debugged it and I'm deep on the  org.antlr.v4.runtime.Lexer class. 

Any ideas ? 

 
{code:java}
== SQL ==bfzugtäeil:string--^^^
at 
org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:306)
   at 
org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:143)
  at 
org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parseDataType(ParseDriver.scala:41)
   at 
com.deutschebahn.zod.fvdl.commons.spark.app.captured.HiveTest2.$anonfun$new$1(HiveTest2.scala:13)
 {code}


> Cannot parse Type from german "umlaut"
> --
>
> Key: SPARK-44108
> URL: https://issues.apache.org/jira/browse/SPARK-44108
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.3.0
>Reporter: Jorge Machado
>Priority: Major
>
> Hello all, 
>  
> I have a client that has a column named : bfzgtäeil
> Spark cannot handle this. My test: 
>  
> {code:java}
> import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
> import org.scalatest.funsuite.AnyFunSuite
> class HiveTest extends AnyFunSuite {
>   test("test that Spark does not cut columns with ä") {
> val data = "bfzugtäeil:string"
> CatalystSqlParser.parseDataType(data)
>   }
> } {code}
> I debugged it and I'm deep on the  org.antlr.v4.runtime.Lexer class. 
> Any ideas ? 
>  
> {code:java}
> == SQL ==bfzugtäeil:string--^^^
>   at 
> org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:306)
>at 
> org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:143)
>   at 
> org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parseDataType(ParseDriver.scala:41)
>at 
> com.deutschebahn.zod.fvdl.commons.spark.app.captured.HiveTest2.$anonfun$new$1(HiveTest2.scala:13)
>  {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-44108) Cannot parse Type from german "umlaut"

[Jorge Machado (Jira)]

 [ 
https://issues.apache.org/jira/browse/SPARK-44108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jorge Machado updated SPARK-44108:
--
Priority: Major  (was: Minor)

> Cannot parse Type from german "umlaut"
> --
>
> Key: SPARK-44108
> URL: https://issues.apache.org/jira/browse/SPARK-44108
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.3.0
>Reporter: Jorge Machado
>Priority: Major
>
> Hello all, 
>  
> I have a client that has a column named : bfzgtäeil
> Spark cannot handle this. My test: 
>  
> {code:java}
> import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
> import org.scalatest.funsuite.AnyFunSuite
> class HiveTest extends AnyFunSuite {
>   test("test that Spark does not cut columns with ä") {
> val data =
>   "bfzugtäeil:string"
> CatalystSqlParser.parseDataType(data)
>   }
> } {code}
> I debugged it and I'm deep on the  org.antlr.v4.runtime.Lexer class. 
> Any ideas ? 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-44108) Cannot parse Type from german "umlaut"

[Jorge Machado (Jira)]

 [ 
https://issues.apache.org/jira/browse/SPARK-44108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jorge Machado updated SPARK-44108:
--
Priority: Minor  (was: Critical)

> Cannot parse Type from german "umlaut"
> --
>
> Key: SPARK-44108
> URL: https://issues.apache.org/jira/browse/SPARK-44108
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 3.3.0
>Reporter: Jorge Machado
>Priority: Minor
>
> Hello all, 
>  
> I have a client that has a column named : bfzgtäeil
> Spark cannot handle this. My test: 
>  
> {code:java}
> import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
> import org.scalatest.funsuite.AnyFunSuite
> class HiveTest extends AnyFunSuite {
>   test("test that Spark does not cut columns with ä") {
> val data =
>   "bfzugtäeil:string"
> CatalystSqlParser.parseDataType(data)
>   }
> } {code}
> I debugged it and I'm deep on the  org.antlr.v4.runtime.Lexer class. 
> Any ideas ? 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-44108) Cannot parse Type from german "umlaut"

[Jorge Machado (Jira)]

Jorge Machado created SPARK-44108:
-

 Summary: Cannot parse Type from german "umlaut"
 Key: SPARK-44108
 URL: https://issues.apache.org/jira/browse/SPARK-44108
 Project: Spark
  Issue Type: Bug
  Components: SQL
Affects Versions: 3.3.0
Reporter: Jorge Machado


Hello all, 

 

I have a client that has a column named : bfzgtäeil

Spark cannot handle this. My test: 

 
{code:java}
import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
import org.scalatest.funsuite.AnyFunSuite

class HiveTest extends AnyFunSuite {

  test("test that Spark does not cut columns with ä") {
val data =
  "bfzugtäeil:string"
CatalystSqlParser.parseDataType(data)
  }

} {code}
I debugged it and I'm deep on the  org.antlr.v4.runtime.Lexer class. 

Any ideas ? 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Unsubscribe

[Jorge Machado]

[jira] [Comment Edited] (SPARK-33772) Build and Run Spark on Java 17

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-33772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653941#comment-17653941
 ] 

Jorge Machado edited comment on SPARK-33772 at 1/3/23 12:27 PM:


I still have an issue with this. Running sbt test fails and I don't know why. 
{code:java}
An exception or error caused a run to abort: class 
org.apache.spark.storage.StorageUtils$ (in unnamed module @0x1aa7ecca) cannot 
access class sun.nio.ch.DirectBuffer (in module java.base) because module 
java.base does not export sun.nio.ch to unnamed module @0x1aa7ecca 
 {code}


was (Author: jomach):
I still have an issue with this. Running sbt test fails and I don't know why. 

[error] Uncaught exception when running .LocalViewCreatorTest: 
java.lang.NoClassDefFoundError: Could not initialize class 
org.apache.spark.storage.StorageUtils$
23/01/03 12:16:40 INFO Utils: Successfully started service 'sparkDriver' on 
port 49902.
[error] sbt.ForkMain$ForkError: java.lang.NoClassDefFoundError: Could not 
initialize class org.apache.spark.storage.StorageUtils$
[error]         at 
org.apache.spark.storage.BlockManagerMasterEndpoint.(BlockManagerMasterEndpoint.scala:114)

> Build and Run Spark on Java 17
> --
>
> Key: SPARK-33772
> URL: https://issues.apache.org/jira/browse/SPARK-33772
> Project: Spark
>  Issue Type: New Feature
>  Components: Build
>Affects Versions: 3.3.0
>Reporter: Dongjoon Hyun
>Assignee: Yang Jie
>Priority: Major
>  Labels: releasenotes
> Fix For: 3.3.0
>
>
> Apache Spark supports Java 8 and Java 11 (LTS). The next Java LTS version is 
> 17.
> ||Version||Release Date||
> |Java 17 (LTS)|September 2021|
> Apache Spark has a release plan and `Spark 3.2 Code freeze` was July along 
> with the release branch cut.
> - https://spark.apache.org/versioning-policy.html
> Supporting new Java version is considered as a new feature which we cannot 
> allow to backport.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-33772) Build and Run Spark on Java 17

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-33772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653941#comment-17653941
 ] 

Jorge Machado edited comment on SPARK-33772 at 1/3/23 12:27 PM:


I still have an issue with this. Running sbt test fails and I don't know why. 

[error] Uncaught exception when running .LocalViewCreatorTest: 
java.lang.NoClassDefFoundError: Could not initialize class 
org.apache.spark.storage.StorageUtils$
23/01/03 12:16:40 INFO Utils: Successfully started service 'sparkDriver' on 
port 49902.
[error] sbt.ForkMain$ForkError: java.lang.NoClassDefFoundError: Could not 
initialize class org.apache.spark.storage.StorageUtils$
[error]         at 
org.apache.spark.storage.BlockManagerMasterEndpoint.(BlockManagerMasterEndpoint.scala:114)


was (Author: jomach):
I still have an issue with this. Running sbt test fails and I don't know why. 

[error] Uncaught exception when running 
com.deutschebahn.zod.fvdl.commons.aws.athena.LocalViewCreatorTest: 
java.lang.NoClassDefFoundError: Could not initialize class 
org.apache.spark.storage.StorageUtils$
23/01/03 12:16:40 INFO Utils: Successfully started service 'sparkDriver' on 
port 49902.
[error] sbt.ForkMain$ForkError: java.lang.NoClassDefFoundError: Could not 
initialize class org.apache.spark.storage.StorageUtils$
[error]         at 
org.apache.spark.storage.BlockManagerMasterEndpoint.(BlockManagerMasterEndpoint.scala:114)

> Build and Run Spark on Java 17
> --
>
> Key: SPARK-33772
> URL: https://issues.apache.org/jira/browse/SPARK-33772
> Project: Spark
>  Issue Type: New Feature
>  Components: Build
>Affects Versions: 3.3.0
>Reporter: Dongjoon Hyun
>Assignee: Yang Jie
>Priority: Major
>  Labels: releasenotes
> Fix For: 3.3.0
>
>
> Apache Spark supports Java 8 and Java 11 (LTS). The next Java LTS version is 
> 17.
> ||Version||Release Date||
> |Java 17 (LTS)|September 2021|
> Apache Spark has a release plan and `Spark 3.2 Code freeze` was July along 
> with the release branch cut.
> - https://spark.apache.org/versioning-policy.html
> Supporting new Java version is considered as a new feature which we cannot 
> allow to backport.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-33772) Build and Run Spark on Java 17

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-33772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17653941#comment-17653941
 ] 

Jorge Machado commented on SPARK-33772:
---

I still have an issue with this. Running sbt test fails and I don't know why. 

[error] Uncaught exception when running 
com.deutschebahn.zod.fvdl.commons.aws.athena.LocalViewCreatorTest: 
java.lang.NoClassDefFoundError: Could not initialize class 
org.apache.spark.storage.StorageUtils$
23/01/03 12:16:40 INFO Utils: Successfully started service 'sparkDriver' on 
port 49902.
[error] sbt.ForkMain$ForkError: java.lang.NoClassDefFoundError: Could not 
initialize class org.apache.spark.storage.StorageUtils$
[error]         at 
org.apache.spark.storage.BlockManagerMasterEndpoint.(BlockManagerMasterEndpoint.scala:114)

> Build and Run Spark on Java 17
> --
>
> Key: SPARK-33772
> URL: https://issues.apache.org/jira/browse/SPARK-33772
> Project: Spark
>  Issue Type: New Feature
>  Components: Build
>Affects Versions: 3.3.0
>Reporter: Dongjoon Hyun
>Assignee: Yang Jie
>Priority: Major
>  Labels: releasenotes
> Fix For: 3.3.0
>
>
> Apache Spark supports Java 8 and Java 11 (LTS). The next Java LTS version is 
> 17.
> ||Version||Release Date||
> |Java 17 (LTS)|September 2021|
> Apache Spark has a release plan and `Spark 3.2 Code freeze` was July along 
> with the release branch cut.
> - https://spark.apache.org/versioning-policy.html
> Supporting new Java version is considered as a new feature which we cannot 
> allow to backport.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Re: Nifi flowfile monitoring

[Jorge Machado]

Activate prometheus exporter and you will have a metric for  number of objects 
in the queue

> On 31. Dec 2022, at 07:08, nayan sharma  wrote:
> 
> Hi Users,
> Is there anyway through which I can monitor or raise alert if any flow file 
> got stuck in nifi queue. 
> 
> For now operation team needs to manually check for these. If you can suggest 
> way through which I can achieve this that would be great.
> 
> Thanks,
> Nayan
> -- 
> Thanks & Regards,
> Nayan Sharma
>  +91-8095382952
> 
>   
> 

Re: Failing to start - keystore properties invalid

[Jorge Machado]

Hi James, 

Can it be that you are trying to start nifi with ssl without authentication ? 
Looks like that.. 

> On 27. Dec 2022, at 22:13, James McMahon  wrote:
> 
> Hello. I am trying to start a secure instance of nifi version 1.16.3. I am 
> getting this error on start attempt:
> 
> 2022-12-27 20:44:21,765 INFO [main] o.a.n.r.v.FileBasedVariableRegistry 
> Loaded a total of 90 properties.  Including precedence overrides effective 
> accessible registry key size is 90
> 2022-12-27 20:44:21,972 WARN [main] o.a.nifi.security.util.SslContextFactory 
> Some keystore properties are populated 
> (/opt/nifi/config_resources/keys/server.jks, , , JKS) but not 
> valid
> 2022-12-27 20:44:21,972 ERROR [main] o.apache.nifi.controller.FlowController 
> Unable to start the flow controller because the TLS configuration was 
> invalid: The keystore properties are not valid
> 2022-12-27 20:44:22,009 ERROR [main] o.s.web.context.ContextLoader Context 
> initialization failed
> org.springframework.beans.factory.BeanCreationException: Error creating bean 
> with name 
> 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration':
>  Initialization of bean failed; nested exception is 
> org.springframework.beans.factory.UnsatisfiedDependencyException: Error 
> creating bean with name 
> 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied dependency 
> expressed through method 'setJwtAuthenticationProvider' parameter 0; nested 
> exception is 
> org.springframework.beans.factory.UnsatisfiedDependencyException: Error 
> creating bean with name 
> 'org.apache.nifi.web.security.configuration.JwtAuthenticationSecurityConfiguration':
>  Unsatisfied dependency expressed through constructor parameter 3; nested 
> exception is org.springframework.beans.factory.BeanCreationException: Error 
> creating bean with name 'flowController': FactoryBean threw exception on 
> object creation; nested exception is java.lang.IllegalStateException: Flow 
> controller TLS configuration is invalid
> 
> 
> 
> This is what my nifi.properties file looks like in this section:
> 
> # security properties #
> nifi.sensitive.props.key=A_KEY_HERE
> nifi.sensitive.props.key.protected=
> nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256
> nifi.sensitive.props.additional.keys=
> 
> nifi.security.autoreload.enabled=false
> nifi.security.autoreload.interval=10 secs
> nifi.security.keystore=/opt/nifi/config_resources/keys/server.jks
> nifi.security.keystoreType=JKS
> nifi.security.keystorePasswd=b0gu5passw0r2!
> nifi.security.keyPasswd=b0gu5passw0r2!
> nifi.security.truststore=/opt/nifi/config_resources/keys/truststore.jks
> nifi.security.truststoreType=JKS
> nifi.security.truststorePasswd=Diff3r3ntBoguspwd#
> nifi.security.user.authorizer=managed-authorizer
> nifi.security.allow.anonymous.authentication=false
> nifi.security.user.login.identity.provider=
> nifi.security.user.jws.key.rotation.period=
> nifi.security.ocsp.responder.url=
> nifi.security.ocsp.responder.certificate=
> 
> I have verified the password for my keystore at the command line (this works):
> 
> sudo keytool -list -v -keystore server.jks
> Enter keystore password: b0gu5passw0r2!
> (I see the result)
> 
> These JKS files were converted by me from a cacert.pem (to truststore.jks) 
> and a server.pfx (for server.jks) using keytool. The cacert.pem and the 
> server.pfx were created by me at TinyCert.org.
> 
> I thought my keyPasswd should be the same as my keystorePasswd, but am I 
> wrong about that? Is it possible that the keyPasswd is the password or 
> passphrase I employed when I created the original server.pfx file?
> 
> What is this error telling me, and how can I fix it?
> 
> To summarize, this is how I got to where I am:
> I created a cacert.pem, an admin.pfx, server.pfx, and client1.pfx using 
> TinyCert.
> While in TinyCert.org I was in with a password and a passphrase.
> I transferred those to my keys directory under my nifi install and used 
> keytool to create a truststore.jks, a server.jks, a client1.jks, and an 
> admin.jks keystore file. 
> Each jks has its own password.
> I can look at the contents of my truststore,jks, my admin.jks, my server.jks, 
> and my client1.jks using keytool, with the password I provided to keytool for 
> admin at the time of conversion.
> 
> Jim
> 
> 
> 

Workflows deployments across environments

[Jorge Machado]

Hello Nifi users, 


Question in the round. Let’s say I have 3 Nifi instances with dev, test and 
prod. What is the recommend way of deploying specific workflows from one 
environment to another ? 
We are using Nifi registry + GitHub as storage for the flows. 

What we want to achieve: 
* in production there should be only read rights
* Deployment in prod should happen when a commit on master or a new git release 
is created. 


I found this: 
https://chamila.dev/blog/2021-07-17_ci-cd-for-apache-nifi-01-intro-and-setup/ 
and 
https://medium.com/tech-at-nordstrom/building-ci-cd-pipelines-with-apache-nifi-and-terraform-acedd653b356
 both ways seems to be like custom made and there is not a standard way of 
doing this. 
Would be great if we had a wiki page on nifi for this. 


Thx
Jorge 

Re: Minifi and ssl config on NiFi

[Jorge Machado]

Oh nice. It is now part of the main Repo ? Did not knew that. Still I found it 
easier to manage two fully nifi instances. On My ansible scripts I had no diff. 

> On 17. Apr 2022, at 17:48, Matt Burgess  wrote:
> 
> MiNiFi is actually alive and well, we just moved it into the NiFi codebase. 
> We’re actively developing a Command-and-Control (C2) capability to remotely 
> update the flow on the agent for example.
> 
> You can configure MiNiFi agents for SSL over Site-to-Site in order to talk to 
> secure NiFi instances. Not sure about the need for a user but you would need 
> to present a certificate the same as you would for connecting to the NiFi UI. 
> Some security features still need to be implemented (like encrypted 
> properties maybe) but you should definitely be able to do what you’re trying 
> to do with MiNiFi, happy to help with any issues you may run into.
> 
> Regards,
> Matt
> 
> 
>> On Apr 17, 2022, at 11:40 AM, Jorge Machado  wrote:
>> 
>> I did this on the pass and I end up switching to Nifi. I think you should 
>> do the same. Minifi is kind of “Dead” not being developed anymore. I found 
>> better to just switch to single instance of nifi 
>> 
>> Regards
>> Jorge 
>> 
>>> On 17. Apr 2022, at 03:30, David Early  wrote:
>>> 
>>> We are considering using several dozen minifi instances to gather data at 
>>> remote locations and send it to a cloud based central NiFi.
>>> 
>>> The problem I am THINKING we will have is setting up ssl. The only way I 
>>> know of to set up ssl for site to site requires a user be configured for 
>>> the incoming data on the destination NiFi and permissions given to that 
>>> user to be able to send data.
>>> 
>>> Am I missing something? Will we have to manually set up a user in the cloud 
>>> NiFi for each minifi instances so we can use ssl transport?
>>> 
>>> Dave
>> 

Re: Minifi and ssl config on NiFi

[Jorge Machado]

I did this on the pass and I end up switching to Nifi. I think you should do 
the same. Minifi is kind of “Dead” not being developed anymore. I found better 
to just switch to single instance of nifi 

Regards
Jorge 

> On 17. Apr 2022, at 03:30, David Early  wrote:
> 
> We are considering using several dozen minifi instances to gather data at 
> remote locations and send it to a cloud based central NiFi.
> 
> The problem I am THINKING we will have is setting up ssl. The only way I know 
> of to set up ssl for site to site requires a user be configured for the 
> incoming data on the destination NiFi and permissions given to that user to 
> be able to send data.
> 
> Am I missing something? Will we have to manually set up a user in the cloud 
> NiFi for each minifi instances so we can use ssl transport?
> 
> Dave

Re: Adding a node to a cluster

[Jorge Machado]

Hey, 

Are you using the zookeeper internally or an external one. I think you need to 
check at least two things: 

* Zookeeper connections. The node needs to register into zookeeper. 
* The node needs to connect to the cluster and replicate the flow files let’s 
say so. 


Overall scaling up is ok, but scaling down it’s complicated you need to make 
sure that you disable the node and remove it from the cluster. You cannot 
simple tear it down as it has state. 

Mit freundlichen Grüßen 
Jorge  

> On 11. Apr 2022, at 15:19, James McMahon  wrote:
> 
> A nifi cluster has been created using a CloudFormation script. The cluster is 
> running, and the customer now wishes to scale up the cluster by adding a 
> node. What is the optimal way to add a node to a running nifi cluster 
> established through CloudFormation? My instinct tells me to do it within the 
> confines of CloudFormation so that the new node isn't lost if CloudFormation 
> is re-run. But does that mean the active cluster needs to be shutdown just to 
> modify the CloudFormation scripts and restart it with the new cluster 
> configuration?
> 
> Has anyone successfully done this, and can offer some tips or a link to an 
> outline of the process?

Re: InvokeHTTP vs invalid SSL certificates

[Jorge Machado]

Just import the certificate into the trust store. 


> On 4. Mar 2022, at 13:59, Jean-Sebastien Vachon  
> wrote:
> 
> Hi all,
> 
> what is the best way to deal with invalid SSL certificates when trying to 
> open an URL using InvokeHTTP?
> 
> 
> Thanks
> 
> Jean-Sébastien Vachon
> Co-Founder & Architect
> Brizo Data, Inc.
> www.brizodata.com 
> 

Re: Round robin load balancing eventually stops using all nodes

[Jorge Machado]

Are you are a dedicated port for transferring the data or using the http 
protocol ? I had similar issues with a remote port connection that got solved 
by not using the http protocol.


> On 3. Sep 2021, at 14:13, Mike Thomsen  wrote:
> 
> We have a 5 node cluster, and sometimes I've noticed that round robin
> load balancing stops sending flowfiles to two of them, and sometimes
> toward the end of the data processing can get as low as a single node.
> Has anyone seen similar behavior?
> 
> Thanks,
> 
> Mike

[jira] [Commented] (KAFKA-5164) SetSchemaMetadata does not replace the schemas in structs correctly

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17391645#comment-17391645
 ] 

Jorge Machado commented on KAFKA-5164:
--

Actually not solved imho. See https://issues.apache.org/jira/browse/KAFKA-7883

> SetSchemaMetadata does not replace the schemas in structs correctly
> ---
>
> Key: KAFKA-5164
> URL: https://issues.apache.org/jira/browse/KAFKA-5164
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.2.1
>Reporter: Ewen Cheslack-Postava
>Assignee: Randall Hauch
>Priority: Major
> Fix For: 0.11.0.0
>
>
> In SetSchemaMetadataTest we verify that the name and version of the schema in 
> the record have been replaced:
> https://github.com/apache/kafka/blob/trunk/connect/transforms/src/test/java/org/apache/kafka/connect/transforms/SetSchemaMetadataTest.java#L62
> However, in the case of Structs, the schema will be attached to both the 
> record and the Struct itself. So we correctly rebuild the Record:
> https://github.com/apache/kafka/blob/trunk/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/SetSchemaMetadata.java#L77
> https://github.com/apache/kafka/blob/trunk/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/SetSchemaMetadata.java#L104
> https://github.com/apache/kafka/blob/trunk/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/SetSchemaMetadata.java#L119
> But if the key or value are a struct, they will still contain the old schema 
> embedded in the struct.
> Ultimately this can lead to validations in other code failing (even for very 
> simple changes like adjusting the name of a schema):
> {code}
> (org.apache.kafka.connect.runtime.WorkerTask:141)
> org.apache.kafka.connect.errors.DataException: Mismatching struct schema
> at io.confluent.connect.avro.AvroData.fromConnectData(AvroData.java:471)
> at io.confluent.connect.avro.AvroData.fromConnectData(AvroData.java:295)
> at 
> io.confluent.connect.avro.AvroConverter.fromConnectData(AvroConverter.java:73)
> at 
> org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:196)
> at 
> org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:167)
> at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:139)
> at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:182)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> The solution to this is probably to check whether we're dealing with a Struct 
> when we use a new schema and potentially copy/reallocate it.
> This particular issue would only appear if we don't modify the data, so I 
> think SetSchemaMetadata is currently the only transformation that would have 
> the issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (KAFKA-7883) Add schema.namespace support to SetSchemaMetadata SMT in Kafka Connect

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/KAFKA-7883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17391583#comment-17391583
 ] 

Jorge Machado commented on KAFKA-7883:
--

I propose to have following: 

new parameter named: 'append.to.current' as boolean

If this parameter is set we just append the namespace to the schema.name()

 

What do you think if yes I would pickup this ticket

> Add schema.namespace support to SetSchemaMetadata SMT in Kafka Connect
> --
>
> Key: KAFKA-7883
> URL: https://issues.apache.org/jira/browse/KAFKA-7883
> Project: Kafka
>  Issue Type: New Feature
>  Components: KafkaConnect
>Affects Versions: 2.1.0
>Reporter: Jérémy Thulliez
>Priority: Minor
>  Labels: features
>
> When using a connector with AvroConverter & SchemaRegistry, users should be 
> able to specify the namespace in the SMT.
> Currently, only "schema.version" and "schema.name" can be specified.
> This is needed because if not specified, generated classes (from avro schema) 
>  are in the default package and not accessible.
> Currently, the workaround is to add a Transformation implementation to the 
> connect classpath.
> It should be native.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (KAFKA-7883) Add schema.namespace support to SetSchemaMetadata SMT in Kafka Connect

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/KAFKA-7883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17391536#comment-17391536
 ] 

Jorge Machado commented on KAFKA-7883:
--

Hey, I think this should not be marked as workaround. Currently if we use this 
it will replace the whole name of the table. If we are using this SMT together 
on the jdbc connector and we have multiple tables it will generate the same 
name for all tables :(  

in teory we should only append to the current name. Example:

 

Avro Object name: "MaterialRaw"

Avro Namespace: "io.dmesh.avro"

 

on the transform imho we should have: 

"transforms.AddNamespace.schema.name": "io.dmesh.avro" which it should generate 
(io.dmesh.avro.MaterialRaw)

 

 

 

> Add schema.namespace support to SetSchemaMetadata SMT in Kafka Connect
> --
>
> Key: KAFKA-7883
> URL: https://issues.apache.org/jira/browse/KAFKA-7883
> Project: Kafka
>  Issue Type: New Feature
>  Components: KafkaConnect
>Affects Versions: 2.1.0
>Reporter: Jérémy Thulliez
>Priority: Minor
>  Labels: features
>
> When using a connector with AvroConverter & SchemaRegistry, users should be 
> able to specify the namespace in the SMT.
> Currently, only "schema.version" and "schema.name" can be specified.
> This is needed because if not specified, generated classes (from avro schema) 
>  are in the default package and not accessible.
> Currently, the workaround is to add a Transformation implementation to the 
> connect classpath.
> It should be native.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Comment Edited] (KAFKA-7883) Add schema.namespace support to SetSchemaMetadata SMT in Kafka Connect

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/KAFKA-7883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17391536#comment-17391536
 ] 

Jorge Machado edited comment on KAFKA-7883 at 8/2/21, 11:58 AM:


Hey, I think this should not be marked as workaround. Currently if we use this 
it will replace the whole name of the table. If we are using this SMT together 
on the jdbc connector and we have multiple tables it will generate the same 
name for all tables :(  

in teory we should only append to the current name. Example:

 

Avro Object name: "MaterialRaw"

Avro Namespace: "io.dmesh.avro"

 

on the transform imho we should have: 

"transforms.AddNamespace.schema.name": "io.dmesh.avro" which it should generate 
(io.dmesh.avro.MaterialRaw) instead of "io.dmesh.avro"

 

 

 


was (Author: jomach):
Hey, I think this should not be marked as workaround. Currently if we use this 
it will replace the whole name of the table. If we are using this SMT together 
on the jdbc connector and we have multiple tables it will generate the same 
name for all tables :(  

in teory we should only append to the current name. Example:

 

Avro Object name: "MaterialRaw"

Avro Namespace: "io.dmesh.avro"

 

on the transform imho we should have: 

"transforms.AddNamespace.schema.name": "io.dmesh.avro" which it should generate 
(io.dmesh.avro.MaterialRaw)

 

 

 

> Add schema.namespace support to SetSchemaMetadata SMT in Kafka Connect
> --
>
> Key: KAFKA-7883
> URL: https://issues.apache.org/jira/browse/KAFKA-7883
> Project: Kafka
>  Issue Type: New Feature
>  Components: KafkaConnect
>Affects Versions: 2.1.0
>Reporter: Jérémy Thulliez
>Priority: Minor
>  Labels: features
>
> When using a connector with AvroConverter & SchemaRegistry, users should be 
> able to specify the namespace in the SMT.
> Currently, only "schema.version" and "schema.name" can be specified.
> This is needed because if not specified, generated classes (from avro schema) 
>  are in the default package and not accessible.
> Currently, the workaround is to add a Transformation implementation to the 
> connect classpath.
> It should be native.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: No Load Balancing since 1.13.2

[Jorge Machado]

Did you tried java 11 ? I have a client running a similar setup to yours but 
with a lower nigh version and it works fine. Maybe it is worth to try it. 


> On 27. Jul 2021, at 12:42, Axel Schwarz  wrote:
> 
> I did indeed, but I updated from u161 to u291, as this was the newest version 
> at that time, because I thought it could help.
> So the issue started under u161. But I just saw that u301 is out. I will try 
> this as well.
> --- Ursprüngliche Nachricht ---
> Von: Pierre Villard 
> Datum: 27.07.2021 10:18:38
> An: users@nifi.apache.org, Axel Schwarz 
> Betreff: Re: No Load Balancing since 1.13.2
> 
> Hi,
> 
> I believe the minor u291 is known to have issues (for some of its early 
> builds). Did you upgrade the Java version recently?
> 
> Thanks,
> Pierre
> 
> Le mar. 27 juil. 2021 à 08:07, Axel Schwarz  > a écrit :
> Dear Community,
> 
> we're running a secured 3 node Nifi Cluster on Java 8_u291 and Debian 7 and 
> experiencing
> problems with load balancing since version 1.13.2.
> 
> I'm fully aware of Issue Nifi-8643 and tested alot around this, but gotta 
> say, that this
> is not our problem. Mainly because the balance port never binds to localhost, 
> but also because I
> implemented all workarounds under version 1.13.2 and even tried version 
> 1.14.0 by now,
> but load blancing still does not work.
> What we experience is best described as "the primary node balances with 
> itself"...
> 
> So what it does is, opening the balancing connections to its own IP instead 
> of the IPs
> of the other two nodes. And the other two nodes don't open balancing 
> connections at all.
> 
> When executing "ss | grep 6342" on the primary node, this is what it looks 
> like:
> 
> [root@nifiHost1 conf]# ss | grep 6342
> tcpESTAB  0  0  192.168.1.10:51380 
> 192.168.1.10:6342 
>  
> tcpESTAB  0  0  192.168.1.10:51376 
> 192.168.1.10:6342 
>  
> tcpESTAB  0  0  192.168.1.10:51378 
> 192.168.1.10:6342 
>  
> tcpESTAB  0  0  192.168.1.10:51370 
> 192.168.1.10:6342 
>  
> tcpESTAB  0  0  192.168.1.10:51372 
> 192.168.1.10:6342 
>  
> tcpESTAB  0  0  192.168.1.10:6342  
> 192.168.1.10:51376    
>  
> tcpESTAB  0  0  192.168.1.10:51374 
> 192.168.1.10:6342 
>  
> tcpESTAB  0  0  192.168.1.10:6342  
> 192.168.1.10:51374    
>  
> tcpESTAB  0  0  192.168.1.10:51366 
> 192.168.1.10:6342 
>  
> tcpESTAB  0  0  192.168.1.10:6342  
> 192.168.1.10:51370    
>  
> tcpESTAB  0  0  192.168.1.10:6342  
> 192.168.1.10:51366    
>  
> tcpESTAB  0  0  192.168.1.10:51368 
> 192.168.1.10:6342 
>  
> tcpESTAB  0  0  192.168.1.10:6342  
> 192.168.1.10:51372    
>  
> tcpESTAB  0  0  192.168.1.10:6342  
> 192.168.1.10:51378    
>  
> tcpESTAB  0  0  192.168.1.10:6342  
> 192.168.1.10:51368    
>  
> tcpESTAB  0  0  192.168.1.10:6342  
> 192.168.1.10:51380 
> 
> Executing it on the other non primary nodes, just returns absolutely nothing.
> 
> Netstat show the following on each server:
> 
> [root@nifiHost1 conf]# netstat -tulpn
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address   Foreign Address State 
>   PID/Program name
> tcp0  0 192.168.1.10:6342   
> 0.0.0.0:*   LISTEN  10352/java
> 
> [root@nifiHost2 conf]# netstat -tulpn
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address   Foreign 

Re: SocketTimeoutExceptions and JVM version

[Jorge Machado]

Is your input configured to be http or binary ? I can recommend using socket 
connection instead of https. You need an extra port but it could be worth. My 
last client we had similar issues. 
Ref: https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html 


Do you see a hanging thread on the input port ? 


> On 4. Jun 2021, at 12:48, Hesselmann, Brian  wrote:
> 
> Hi all,
> 
> We've been running into some problems with keeping our Nifi stable. We're 
> running the 'official' apache/nifi:1.13.2 docker image openjdk 1.8.0_282 on a 
> single node without clustering. 
> 
> Our instance receives about 5000requests per minute and after running 
> smoothly for 8-12 hours we start seeing a lot of SocketTimeoutExceptions. I 
> saw mentioned before that certain JVM versions have issues with 
> SocketTimeouts, so was curious which versions have this issue so we can try 
> upgrading to a different one.
> 
> Best,
> Brian

[jira] [Commented] (MINIFI-546) Upgrade MiNiFi to Java 11

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/MINIFI-546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17340560#comment-17340560
 ] 

Jorge Machado commented on MINIFI-546:
--

we actually stopped using the minifi because of the lacking of a proper UI 

> Upgrade MiNiFi to Java 11
> -
>
> Key: MINIFI-546
> URL: https://issues.apache.org/jira/browse/MINIFI-546
> Project: Apache NiFi MiNiFi
>  Issue Type: Wish
>Affects Versions: 0.5.0
>    Reporter: Jorge Machado
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Minifi only runs on Java 8 which is not support anymore. We should start 
> preparing an upgrade. 
> Issues to discuss: 
> * jaxb -> was removed from java SDK so we need to use Jackson or add the jaxb 
> dependency. 
> * Certificate issues due to new api



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: RecordPath...With INNER records

[Jorge Machado]

Hey Greene, 

The LookupRecord as a RecordPath as input. Check out this docs: 
https://nifi.apache.org/docs/nifi-docs/html/record-path-guide.html#structure 
 
or https://www.nifi.rocks/record-path-cheat-sheet/ 


In your case you could enter /outerrecord/[0]/key. Something like that, I did 
not test it but you get the idea. 
Regards
Jorge from Datamesh 
www.dmesh.io


> On 24. Mar 2021, at 21:59, Greene (US), Geoffrey N 
>  wrote:
> 
> Sorry for the resend. Thought I’d try one more time.  I’m struggling with 
> LookupRecord.
>  
> I’m making good progress leaning to use using Lookup Record  Processor.
> I have it working so that I know how to turn
> [{“key”: “value1”}, {“key”:”value2”}, {“key”:”value3”}]
> Into
> [{“key”: “value1”,”enhanced”:1}, {“key”:”value2”,”enhanced”:2}, 
> {“key”:”value3”,”enhanced”:3}]
> BUT
> What if your array is actually inside an outer record? Is that supported?  
> How?
>  
> What if you have 
> {“foo”:”bar”, “outerrecord”:[{“key”: “value1”}, {“key”:”value2”}, 
> {“key”:”value3”}]}
> And want to turn it into
> {“foo”:”bar”, “outerrecord”:[{“key”: “value1”,”enhanced”:1}, 
> {“key”:”value2”,”enhanced”:2}, {“key”:”value3”,”enhanced”:3}]}
> How do you specify your lookup key is outerrecord/key, but the array of 
> records is actually located at /outerrecord.  Is there some kind of path 
> magic? Or is there some kind of flow magic where you have to separate 
> outerrecord into a separate flow file, and then (?somehow?) glue it back in 
> after its processed?
>  
> (If necessary, I could certainly put the whole thing into an array too, of 
> course, if that helps.)
>  
> Thanks! 

[jira] [Commented] (AIRFLOW-5655) Incorrect capitalization of env var causes task start to fail

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296355#comment-17296355
 ] 

Jorge Machado commented on AIRFLOW-5655:


I can confirm that I had the same error. Fixed by double checking all vars

> Incorrect capitalization of env var causes task start to fail
> -
>
> Key: AIRFLOW-5655
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5655
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.10.4
> Environment: Tested on Linux (Ubuntu xenial)
>Reporter: Marius Shekow
>Priority: Major
>
> Suppose you define an environment variable such as
> {code:java}
> AIRFLOW__SMTP__SMTP_port=25{code}
> where PORT was accidentally written in lower-case. When running "airflow 
> scheduler" this will cause the following stack-trace, which is not indicative 
> of the cause.
> {noformat}
> [2019-10-14 13:32:23,666] {cli.py:516} {cli.py:516} INFO - Running 
>  [queued]> on host ubuntu-xenial
> Traceback (most recent call last):
>   File "/usr/local/bin/airflow", line 32, in 
> args.func(args)
>   File "/usr/local/lib/python3.5/dist-packages/airflow/utils/cli.py", line 
> 74, in wrapper
> return f(*args, **kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 522, 
> in run
> _run(args, dag, ti)
>   File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 435, 
> in _run
> run_job.run()
>   File "/usr/local/lib/python3.5/dist-packages/airflow/jobs/base_job.py", 
> line 213, in run
> self._execute()
>   File 
> "/usr/local/lib/python3.5/dist-packages/airflow/jobs/local_task_job.py", line 
> 75, in _execute
> self.task_runner = get_task_runner(self)
>   File 
> "/usr/local/lib/python3.5/dist-packages/airflow/task/task_runner/__init__.py",
>  line 38, in get_task_runner
> return StandardTaskRunner(local_task_job)
>   File 
> "/usr/local/lib/python3.5/dist-packages/airflow/task/task_runner/standard_task_runner.py",
>  line 31, in __init__
> super(StandardTaskRunner, self).__init__(local_task_job)
>   File 
> "/usr/local/lib/python3.5/dist-packages/airflow/task/task_runner/base_task_runner.py",
>  line 93, in __init__
> include_cmds=False)
>   File 
> "/usr/local/lib/python3.5/dist-packages/airflow/utils/configuration.py", line 
> 35, in tmp_configuration_copy
> cfg_dict = conf.as_dict(display_sensitive=True, raw=True)
>   File "/usr/local/lib/python3.5/dist-packages/airflow/configuration.py", 
> line 436, in as_dict
> opt = opt.replace('%', '%%')
> AttributeError: 'NoneType' object has no attribute 'replace'
> [2019-10-14 13:32:23,864] {sequential_executor.py:52} ERROR - Failed to 
> execute task Command '['airflow', 'run', 'example_bash_operator', 
> 'run_this_last', '2019-10-13T00:00:00+00:00', '--local', '--pool', 
> 'default_pool', '-sd', 
> '/usr/local/lib/python3.5/dist-packages/airflow/example_dags/example_bash_operator.py']'
>  returned non-zero exit status 1.
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AIRFLOW-5655) Incorrect capitalization of env var causes task start to fail

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/AIRFLOW-5655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17296337#comment-17296337
 ] 

Jorge Machado commented on AIRFLOW-5655:


I'm getting the same issue. 
{code:java}
File 
"/home/airflow/.local/lib/python3.7/site-packages/airflow/task/task_runner/standard_task_runner.py",
 line 37, in __init__
super(StandardTaskRunner, self).__init__(local_task_job)
File 
"/home/airflow/.local/lib/python3.7/site-packages/airflow/task/task_runner/base_task_runner.py",
 line 91, in __init__
include_cmds=False)
File 
"/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/configuration.py",
 line 35, in tmp_configuration_copy
cfg_dict = conf.as_dict(display_sensitive=True, raw=True)
File 
"/home/airflow/.local/lib/python3.7/site-packages/airflow/configuration.py", 
line 562, in as_dict
opt = opt.replace('%', '%%')
AttributeError: 'NoneType' object has no attribute 'replace'
[2021-03-05 20:53:49,697: ERROR/ForkPoolWorker-1] execute_command encountered a 
CalledProcessError
Traceback (most recent call last):
File 
"/home/airflow/.local/lib/python3.7/site-packages/airflow/executors/celery_executor.py",
 line 78, in execute_command
close_fds=True, env=env)
File "/usr/local/lib/python3.7/subprocess.py", line 363, in check_call
raise CalledProcessError(retcode, cmd)
{code}
Use helm chart  7.16.0

> Incorrect capitalization of env var causes task start to fail
> -
>
> Key: AIRFLOW-5655
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5655
> Project: Apache Airflow
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.10.4
> Environment: Tested on Linux (Ubuntu xenial)
>Reporter: Marius Shekow
>Priority: Major
>
> Suppose you define an environment variable such as
> {code:java}
> AIRFLOW__SMTP__SMTP_port=25{code}
> where PORT was accidentally written in lower-case. When running "airflow 
> scheduler" this will cause the following stack-trace, which is not indicative 
> of the cause.
> {noformat}
> [2019-10-14 13:32:23,666] {cli.py:516} {cli.py:516} INFO - Running 
>  [queued]> on host ubuntu-xenial
> Traceback (most recent call last):
>   File "/usr/local/bin/airflow", line 32, in 
> args.func(args)
>   File "/usr/local/lib/python3.5/dist-packages/airflow/utils/cli.py", line 
> 74, in wrapper
> return f(*args, **kwargs)
>   File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 522, 
> in run
> _run(args, dag, ti)
>   File "/usr/local/lib/python3.5/dist-packages/airflow/bin/cli.py", line 435, 
> in _run
> run_job.run()
>   File "/usr/local/lib/python3.5/dist-packages/airflow/jobs/base_job.py", 
> line 213, in run
> self._execute()
>   File 
> "/usr/local/lib/python3.5/dist-packages/airflow/jobs/local_task_job.py", line 
> 75, in _execute
> self.task_runner = get_task_runner(self)
>   File 
> "/usr/local/lib/python3.5/dist-packages/airflow/task/task_runner/__init__.py",
>  line 38, in get_task_runner
> return StandardTaskRunner(local_task_job)
>   File 
> "/usr/local/lib/python3.5/dist-packages/airflow/task/task_runner/standard_task_runner.py",
>  line 31, in __init__
> super(StandardTaskRunner, self).__init__(local_task_job)
>   File 
> "/usr/local/lib/python3.5/dist-packages/airflow/task/task_runner/base_task_runner.py",
>  line 93, in __init__
> include_cmds=False)
>   File 
> "/usr/local/lib/python3.5/dist-packages/airflow/utils/configuration.py", line 
> 35, in tmp_configuration_copy
> cfg_dict = conf.as_dict(display_sensitive=True, raw=True)
>   File "/usr/local/lib/python3.5/dist-packages/airflow/configuration.py", 
> line 436, in as_dict
> opt = opt.replace('%', '%%')
> AttributeError: 'NoneType' object has no attribute 'replace'
> [2019-10-14 13:32:23,864] {sequential_executor.py:52} ERROR - Failed to 
> execute task Command '['airflow', 'run', 'example_bash_operator', 
> 'run_this_last', '2019-10-13T00:00:00+00:00', '--local', '--pool', 
> 'default_pool', '-sd', 
> '/usr/local/lib/python3.5/dist-packages/airflow/example_dags/example_bash_operator.py']'
>  returned non-zero exit status 1.
> {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: NiFI fronend disconnects after some time

[Jorge Machado]

How much memory do you have on your instance?
I would say the fetch size should be bigger  then the Max Rows… 


> On 1. Mar 2021, at 06:37, Vibhath Ileperuma  
> wrote:
> 
> Hi all,
> 
> I tried to fetch a large dataset from postgresql using the ExcureSQL 
> processor. 
> The configurations I used is,
> Fetch Size : 100
> Max Rows per flow file 20
> Output Batch Size 1
> 
> When I start the processor, after some time, NIFI web frontend disconnects. I 
> checked the app log, but there were no errors. 
> 
> I would be grateful if you could suggest to me the reason for this issue and 
> a way to overcome this.
>   
> Thanks & Regards
> Vibhath Ileperuma
> 

Re: Feature requests for Mesos

[Jorge Machado]

Hi Samuel, 

To be honest, I would not invest any more Time on Mesos. The features from 
Kubernetes are just way better. :) 

> On 28. Feb 2021, at 12:54, Samuel Marks  wrote:
> 
> Decouple Apache ZooKeeper, enabling Apache Mesos to run completely without
> ZooKeeper. Specifically enable a choice between ZooKeeper, etcd, and consul.
> 
> My organisation is somewhat interested in contributing this. We tried in
> the past but came across some hurdles on the Mesos organisation end. Open
> to trying again, but will need a clear pathway to getting this accepted.
> 
> Samuel Marks
> Charity  | consultancy 
> | open-source  | LinkedIn
> 
> 
> 
> On Sun, Feb 28, 2021 at 7:39 PM Qian Zhang  wrote:
> 
>> Hi Folks,
>> 
>> To reboot this awesome project, I'd like to collect feature requests for
>> Mesos. Please let us know your requirements for Mesos and whether you or
>> your organization would like to contribute to the implementation of the
>> requirements. Thanks!
>> 
>> 
>> Regards,
>> Qian Zhang
>> 

Change internal state of a processor

[Jorge Machado]

Hello everyone, 

I think we need a tool that allows us to change the internal state of a 
processor. Like for example I would like to got 10 steps back from the state 
that has a value of 10. The resulting state would be 0
We could just overwrite what is there…


Thanks 

[jira] [Updated] (NIFI-8227) Enable extend of AbstractDatabaseFetchProcessor without adding the whole nifi-standard processors

[Jorge Machado (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-8227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jorge Machado updated NIFI-8227:

Affects Version/s: 1.12.1

> Enable extend of AbstractDatabaseFetchProcessor without adding the whole 
> nifi-standard processors
> -
>
> Key: NIFI-8227
> URL: https://issues.apache.org/jira/browse/NIFI-8227
> Project: Apache NiFi
>  Issue Type: Wish
>Affects Versions: 1.12.1
>    Reporter: Jorge Machado
>Priority: Minor
>
> Hi, 
>  
> I'm trying to extend AbstractDatabaseFetchProcessor, which ends up to be 
> difficult. The only way is to pull the standard processors as dependency and 
> this will duplicate all processors on the canvas. We should refactor so that 
> AbstractDatabaseFetchProcessor is moved to a place where can be extended 
> properlly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (NIFI-8227) Enable extend of AbstractDatabaseFetchProcessor without adding the whole nifi-standard processors

[Jorge Machado (Jira)]

 [ 
https://issues.apache.org/jira/browse/NIFI-8227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jorge Machado updated NIFI-8227:

Priority: Minor  (was: Major)

> Enable extend of AbstractDatabaseFetchProcessor without adding the whole 
> nifi-standard processors
> -
>
> Key: NIFI-8227
> URL: https://issues.apache.org/jira/browse/NIFI-8227
> Project: Apache NiFi
>  Issue Type: Wish
>    Reporter: Jorge Machado
>Priority: Minor
>
> Hi, 
>  
> I'm trying to extend AbstractDatabaseFetchProcessor, which ends up to be 
> difficult. The only way is to pull the standard processors as dependency and 
> this will duplicate all processors on the canvas. We should refactor so that 
> AbstractDatabaseFetchProcessor is moved to a place where can be extended 
> properlly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (NIFI-8227) Enable extend of AbstractDatabaseFetchProcessor without adding the whole nifi-standard processors

[Jorge Machado (Jira)]

Jorge Machado created NIFI-8227:
---

 Summary: Enable extend of AbstractDatabaseFetchProcessor without 
adding the whole nifi-standard processors
 Key: NIFI-8227
 URL: https://issues.apache.org/jira/browse/NIFI-8227
 Project: Apache NiFi
  Issue Type: Wish
Reporter: Jorge Machado


Hi, 

 

I'm trying to extend AbstractDatabaseFetchProcessor, which ends up to be 
difficult. The only way is to pull the standard processors as dependency and 
this will duplicate all processors on the canvas. We should refactor so that 
AbstractDatabaseFetchProcessor is moved to a place where can be extended 
properlly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Detect duplicate record reader

[Jorge Machado]

Hey Jeremy, 

Something linke this 
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.5.0/org.apache.nifi.processors.standard.DetectDuplicate/index.html
 

 ? 


> On 15. Feb 2021, at 04:45, Jeremy Pemberton-Pigott  
> wrote:
> 
> Hi everyone, I'm wondering if there is a Detect Duplicate processor that can 
> read records from a flow file and as output gives just the non-duplicates 
> (can be single records or a group of non-duplicates would be better).  I want 
> to use a record reader to avoid splitting the json content into 1s of 
> flow files to detect the duplicates.  Immediately after this flow is a record 
> reader/writer going to HBase.
> 
> Jeremy

GenerateTableFetch Question

[Jorge Machado]

Hey again everyone, 

Is it possible that the GenerateTableFetch passes the max value into the table 
name field before it executes it ? 

Background for this I have a stored procedure that I pass in as table name. 
This needs two parameters a min and a max. Where min should be the maximum 
value that I have on the state. 

Thanks 

Re: How to proper use DistributedCacheServer ?

[Jorge Machado]

Thanks to all ! 
I was thinking about hazelcast hashmap 


> On 12. Feb 2021, at 16:08, Chris Sampson  wrote:
> 
> I'm pretty sure they don't, you need to use an external implementation, such 
> as Redis.
> 
> The DistributedMapCacheServer is a reference implementation but there are 
> alternatives - see the list of available implementations in the "Distributed 
> Cache Service" property of the PutDistributedMapCache processor docs [1].
> 
> You can also implement your own by extending the DistributedCacheServer class.
> 
> 
> [1]: 
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.12.1/org.apache.nifi.processors.standard.PutDistributedMapCache/index.html
>  
> <https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.12.1/org.apache.nifi.processors.standard.PutDistributedMapCache/index.html>
> 
> ---
> Chris Sampson
> IT Consultant
> chris.samp...@naimuri.com <mailto:chris.samp...@naimuri.com>
>  <https://www.naimuri.com/>
> 
> 
> On Fri, 12 Feb 2021 at 14:48, Jorge Machado  <mailto:jom...@me.com>> wrote:
> Hey everyone, 
> 
> Is there any documentation on how to use DistributedCacheServer ? Currently 
> from what I see this is single point of failure  or does it really sync the 
> data between nodes ? 
> 
> I want to have something similar to zookeeper state but not in zookeeper 
> because it needs to be available between processors. My rough problem: 
> 
> 
> Flow : 
> 
> *  Start the flow and store a state (Similar to the QueryTable, which uses 
> zookeeper to store the max row)
> *  do some processing
> *  update the state
> 
> 
> This needs to be reachable between servers of course. What I tested the 
> DistributedMapCacheClientService  needs a server to connect which I point to 
> localhost. But if the entry that I need is on another server how do they sync 
> the data ? 
> 
> Thx

How to proper use DistributedCacheServer ?

[Jorge Machado]

Hey everyone, 

Is there any documentation on how to use DistributedCacheServer ? Currently 
from what I see this is single point of failure  or does it really sync the 
data between nodes ? 

I want to have something similar to zookeeper state but not in zookeeper 
because it needs to be available between processors. My rough problem: 


Flow : 

*  Start the flow and store a state (Similar to the QueryTable, which uses 
zookeeper to store the max row)
*  do some processing
*  update the state


This needs to be reachable between servers of course. What I tested the 
DistributedMapCacheClientService  needs a server to connect which I point to 
localhost. But if the entry that I need is on another server how do they sync 
the data ? 

Thx 

Re: [E] After upgrade to 1.11.4, flowController fails to start due to invalid clusterCoordinator port 0

[Jorge Machado]

For cluster mode check the configs that are on xml files. I had similar issues 
when I did not define the values. Letting them empty makes issues. 
Best regards
Jorge CEO of Datamesh GmbH (www.dmesh.io)

> On 9. Feb 2021, at 02:19, Pat White  wrote:
> 
> Thanks very much for the feedback Joe, much appreciated. Checking as you 
> suggested, nothing yet but that's got to be it, some config issue on my part, 
> that's messing up property parsing.
> 
> 
> 
> 
> 
> On Mon, Feb 8, 2021 at 5:15 PM Joe Witt  > wrote:
> PatW
> 
> I'd triple-check to ensure there are no weird/special/unexpected characters 
> in your nifi.properties file.  These are often not obvious in default text 
> views so you might need to explicitly set some view to expose them.
> 
> Yeah this is certainly not a great user experience - we give you just enough 
> to have an idea but leave plenty to the imagination here.  
> 
> I suppose the good news is we know it is a port.
> 
> Check lines in/around 
> nifi.remote.input.socket.port=
> nifi.web.http.port=8080
> nifi.web.https.port=
> nifi.cluster.node.protocol.port=
> nifi.cluster.load.balance.port=6342
> 
> Thanks
> 
> On Mon, Feb 8, 2021 at 2:47 PM Pat White  > wrote:
> Hi Folks,
> 
> Appreciate any debugging help on a very odd error, after upgrading a Nifi 
> cluster from 1.6.0 to 1.11.4, flowController fails to start due to:
> 
> Caused by: org.springframework.beans.factory.BeanCreationException: Error 
> creating bean with name 'flowService': FactoryBean threw exception on object 
> creation; nested exception is 
> org.springframework.beans.factory.BeanCreationException: Error creating bean 
> with name 'flowController' defined in class path resource [nifi-context.xml]: 
> Cannot resolve reference to bean 'clusterCoordinator' while setting bean 
> property 'clusterCoordinator'; nested exception is 
> org.springframework.beans.factory.BeanCreationException: Error creating bean 
> with name 'clusterCoordinator': FactoryBean threw exception on object 
> creation; nested exception is java.lang.IllegalArgumentException: Port must 
> be inclusively in the range [1, 65535].  Port given: 0
> 
> 
> The error trace is very similar to the example Andy described in NIFI-6336, 
> the issue there i believe is not specifying 'nifi.cluster.node.protocol.port' 
> in 'nifi.properties', however my conf has that set, 
> 'nifi.cluster.node.protocol.port=50233'  and should be using 50233 instead of 
> '0'.
> 
> Cluster had been running fine previously and as far as i can tell, Nifi and 
> ZK confs and settings are all ok. Also compared to another cluster that had 
> been upgraded with no issues, and is running 1.11.4 just fine.
> 
> Increased debug logging but without success so far. Am looking at the right 
> property association?
> 
> patw
> 
> 

Re: NIFI - Performance issues

[Jorge Machado]

Another thing to look would be to check if you are creating two much flow files 
as this hammers the disks. If you see very high memory usage it could be you 
are having to much data in attributes and not in the content of the flow file. 

> On 8. Feb 2021, at 07:07, nathan.engl...@bt.com wrote:
> 
> Hi Asmath,
>  
> In terms of performance tuning, I’ve found a lot that needs to be considered.
>  
> I’ve done tuning it’s mainly been caused large queues building due to the 
> read-write queues on the content repo disks being very high. I’ve also had 
> previous threads on the JOLT default transformation, which should be avoided 
> at all costs as it’s not performant.
>  
> If you are confident the performance bottleneck isn’t the hardware in its 
> cluster itself, I would look at of tuning threads on the flow. I can 
> recommend looking at Mark Payne’s NiFi Notes Videos on Youtube: 
> https://www.youtube.com/channel/UCcNsK56MZRXzPbRfe8sODWw 
> 
>  
> Bear in mind, that you may need to go round the loop of tuning the hardware 
> and the flow to meet your requirements.
>  
> Kind Regards,
>  
> Nathan
> From: KhajaAsmath Mohammed [mailto:mdkhajaasm...@gmail.com] 
> Sent: 08 February 2021 08:22
> To: users@nifi.apache.org
> Subject: NIFI - Performance issues
>  
> Hi,
>  
> I am looking for some information on how to improve performance of our flows. 
> Any suggestions?
>  
> 1. how to troubleshoot which processor in the nifi is causing an issue?
> 2. What is the best way to increase the threads or tasks for a particular 
> processor?
> 3. How to debug the flow when there is an issue on a particular processor 
> group? 
>  
> Thanks,
> Asmath

[jira] [Created] (MINIFI-546) Upgrade minifi to java 11 and nifi 1.12

[Jorge Machado (Jira)]

Jorge Machado created MINIFI-546:


 Summary: Upgrade minifi to java 11 and nifi 1.12
 Key: MINIFI-546
 URL: https://issues.apache.org/jira/browse/MINIFI-546
 Project: Apache NiFi MiNiFi
  Issue Type: Wish
Affects Versions: 0.5.0
Reporter: Jorge Machado


Minifi only runs on Java 8 which is not support anymore. We should start 
preparing an upgrade. 

Issues to discuss: 

* jaxb -> was removed from java SDK so we need to use Jackson or add the jaxb 
dependency. 

* Certificate issues due to new api



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Subject: [RESULT][VOTE] Release Apache Mesos 1.11.0 (rc1)

[Jorge Machado]

Unsubscribe


> On 24. Nov 2020, at 15:22, Andrei Sekretenko  wrote:
> 
> Hi all,
> 
> The vote for Mesos 1.11.0 (rc1) has passed with the
> following votes.
> 
> +1 (Binding)
> --
> Vinod Kone
> Till Toenshoff
> Qian Zhang
> Andrei Sekretenko
> 
> There were no 0 or -1 votes.
> 
> Please find the release at:
> https://dist.apache.org/repos/dist/release/mesos/1.11.0
> 
> It is recommended to use a mirror to download the release:
> http://www.apache.org/dyn/closer.cgi
> 
> The CHANGELOG for the release is available at:
> https://gitbox.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.11.0
> 
> The mesos-1.11.0.jar has been released to:
> https://repository.apache.org
> 
> The website (http://mesos.apache.org) will be updated shortly to
> reflect this release.
> 
> Thanks,
> Andrei Sekretenko

[jira] [Commented] (AVRO-2890) java JSON decoder does not respect default values for fields

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/AVRO-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185861#comment-17185861
 ] 

Jorge Machado commented on AVRO-2890:
-

I have the same issue

> java JSON decoder does not respect default values for fields
> 
>
> Key: AVRO-2890
> URL: https://issues.apache.org/jira/browse/AVRO-2890
> Project: Apache Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.10.0
>Reporter: Sharath Avadoot Gururaj
>Priority: Major
>
> Consider the following schema:
> {code:java}
> {"namespace": "example.avro",
>  "type": "record",
>  "name": "Nic",
>  "fields": [
>{"name" : "ip", "type" : "string", "default" : ""}
>  ]
> }
> and the following empty json{code}
> {code:java}
> {}{code}
> I expect that parsing is successful with this code
> {code:java}
> public void jsonToAvro() throws Exception {
> JsonParser parser;
> Schema schema = new 
> Schema.Parser().parse(readClasspathFile(s.schema));
> Decoder decoder;
> JsonFactory factory = new JsonFactory();
> if(s.linesep) {
> parser = 
> factory.createParser(Files.newInputStream(Paths.get(s.input)));
> decoder = DecoderFactory.get().jsonDecoder(schema, 
> Files.newInputStream(Paths.get(s.input)));
> } else {
> parser = 
> factory.createParser(Files.readAllBytes(Paths.get(s.input)));
> decoder = DecoderFactory.get().jsonDecoder(schema, new 
> String(Files.readAllBytes(Paths.get(s.input;
> }
> parser.configure(JsonParser.Feature.INCLUDE_SOURCE_IN_LOCATION, 
> true);
> //Decoder decoder = new ExtendedJsonDecoder(schema, parser, true );
> DataFileWriter writer;
> CountingOutputStream output = new 
> CountingOutputStream(Files.newOutputStream(Paths.get(s.output)));
> DatumReader reader = new 
> GenericDatumReader<>(schema);
> writer = new DataFileWriter<>(new GenericDatumWriter<>());
> writer.create(schema, output);
> //Decoder decoder = new ExtendedJsonDecoder(schema, parser, true );
> GenericRecord datum = null;
> while (true) {
> try {
> datum = reader.read(datum, decoder);
> } catch (EOFException eofe) {
> break;
> }
> writer.append(datum);
> }
> writer.flush();
> }
> {code}
> But I get the following error
>  
> {noformat}
> org.apache.avro.AvroTypeException: Expected field name not found: ip
>   at org.apache.avro.io.JsonDecoder.doAction(JsonDecoder.java:473) 
> ~[avro-1.10.0.jar:1.10.0]
>   at org.apache.avro.io.parsing.Parser.advance(Parser.java:86) 
> ~[avro-1.10.0.jar:1.10.0]
>   at org.apache.avro.io.JsonDecoder.advance(JsonDecoder.java:132) 
> ~[avro-1.10.0.jar:1.10.0]
>   at org.apache.avro.io.JsonDecoder.readString(JsonDecoder.java:212) 
> ~[avro-1.10.0.jar:1.10.0]
>   at org.apache.avro.io.JsonDecoder.readString(JsonDecoder.java:207) 
> ~[avro-1.10.0.jar:1.10.0]
>   at 
> org.apache.avro.io.ResolvingDecoder.readString(ResolvingDecoder.java:208) 
> ~[avro-1.10.0.jar:1.10.0]
>   at 
> org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:469)
>  ~[avro-1.10.0.jar:1.10.0]
>   at 
> org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:459)
>  ~[avro-1.10.0.jar:1.10.0]
>   at 
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:191)
>  ~[avro-1.10.0.jar:1.10.0]
>   at 
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160) 
> ~[avro-1.10.0.jar:1.10.0]
>   at 
> org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:259)
>  ~[avro-1.10.0.jar:1.10.0]
>   at 
> org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
>  ~[avro-1.10.0.jar:1.10.0]
>   at 
> org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
>  ~[avro-1.10.0.jar:1.10.0]
>   at 
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160) 
> ~[avro-1.10.0.jar:1.10.0]
>   at 
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153) 
> ~[avro-1.10.0.jar:1.10.0]
>   at sha.Deser.jsonToAvro(Deser.java:101) ~[classes/:?]
>   at sha.Deser.go(Deser.java:70) ~[classes/:?]
>   at sha.Deser.main(Deser.java:43) [classes/:?]
> {noformat}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (AVRO-1582) Json serialization of nullable fileds and fields with default values improvement.

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/AVRO-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185858#comment-17185858
 ] 

Jorge Machado commented on AVRO-1582:
-

I think I'm hitting similar things with avro 1.10.

 
{code:java}
Exception in thread "main" org.apache.avro.AvroTypeException: Expected long. 
Got VALUE_STRINGException in thread "main" org.apache.avro.AvroTypeException: 
Expected long. Got VALUE_STRING at 
org.apache.avro.io.JsonDecoder.error(JsonDecoder.java:511) at 
org.apache.avro.io.JsonDecoder.readLong(JsonDecoder.java:177) at 
org.apache.avro.io.ResolvingDecoder.readLong(ResolvingDecoder.java:169) at 
org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:197)
 at 
org.apache.avro.specific.SpecificDatumReader.readField(SpecificDatumReader.java:136)
 at 
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
 at 
org.apache.avro.specific.SpecificDatumReader.readRecord(SpecificDatumReader.java:123)
 at 
org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
 at 
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160) at 
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
{code}

> Json serialization of nullable fileds and fields with default values 
> improvement.
> -
>
> Key: AVRO-1582
> URL: https://issues.apache.org/jira/browse/AVRO-1582
> Project: Apache Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.0
>Reporter: Zoltan Farkas
>Priority: Major
> Attachments: AVRO-1582-PATCH
>
>
> Currently serializing a nullable field of type union like:
> "type" : ["null","some type"]
> when serialized as JSON results in:  
> "field":{"some type":"value"}
> when it could be:
> "field":"value"
> Also fields that equal the the default value can be omitted from the 
> serialized data. This is possible because the reader will have the writer's 
> schema and can infer the field values. This reduces the size of the json 
> messages.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: External Access using InvokeHTTP_Test processor and StandardSSLContextService

[Jorge Machado]

Hi Dan, 

Seems like this is a jvm issue. 

Try this: 
https://confluence.atlassian.com/kb/unable-to-connect-to-ssl-services-due-to-pkix-path-building-failed-error-779355358.html
 

Diagnosis
Use SSL Poke to verify connectivity
Try the Java class SSLPoke to see if your truststore contains the right 
certificates. This will let you connect to a SSL service, send a byte of input, 
and watch the output.
Download SSLPoke.class 

Execute the class as per the below, changing the URL and port appropriately. 
Take care that you are running the same Java as what Confluence is running 
with. If you used the installer you will need to use /jre/java
$JAVA_HOME/bin/java SSLPoke jira.example.com 443
A mail server may be mail.example.com  465 .

The jira.example.com  is your custom site. Add the CA 
Certs 



> On 6. Aug 2020, at 14:08,  
>  wrote:
> 
> It tells you most probably that the CA cert from the remote HTTPS server 
> hasn’t been found in the truststore you’ve defined to access the site. So 
> please check again the CA cert and the truststore…
>  
> Cheers Josef
>  
>  
> From: "White, Daniel" mailto:daniel.wh...@lgim.com>>
> Reply to: "users@nifi.apache.org " 
> mailto:users@nifi.apache.org>>
> Date: Thursday, 6 August 2020 at 13:07
> To: "users@nifi.apache.org" 
> Subject: External Access using InvokeHTTP_Test processor and 
> StandardSSLContextService
>  
> Confidential
>  
> Hi All,
>  
> We’ve setup the truststore from the NiFi processor. However we get the 
> following error when trying to connect to an external HTTPS location
>  
> The error I get is: PKIX path building failed: 
> sun.security.provider.certpath.SunCertPathBuilderException: unable to find 
> valid certification path to requested target
>  
> Any ideas? Assume this is a cert issue on the Nifi server.
>  
> Thanks
>  
> Dan White 
> Lead Technical Architect
> Legal & General Investment Management
> One Coleman Street, London, EC2R 5AA
> Tel: +44 203 124 4048
> Mob: +44 7980 027 656
> www.lgim.com 
>  
> This e-mail (and any attachments) may contain privileged and/or confidential 
> information. If you are not the intended recipient please do not disclose, 
> copy, distribute, disseminate or take any action in reliance on it. If you 
> have received this message in error please reply and tell us and then delete 
> it. Should you wish to communicate with us by e-mail we cannot guarantee the 
> security of any data outside our own computer systems. 
> 
> Any information contained in this message may be subject to applicable terms 
> and conditions and must not be construed as giving investment advice within 
> or outside the United Kingdom or Republic of Ireland. 
> 
> Telephone Conversations may be recorded for your protection and to ensure 
> quality of service 
> 
> Legal & General Investment Management Limited (no 2091894), LGIM Real Assets 
> (Operator) Limited (no 05522016), LGIM (International) Limited (no 7716001) 
> Legal & General Unit Trust Managers (no 1009418), GO ETF Solutions LLP 
> (OC329482) and LGIM Corporate Director Limited (no 7105051) are authorised 
> and regulated by the Financial Conduct Authority. All are registered in 
> England & Wales with a registered office at One Coleman Street, London, EC2R 
> 5AA 
> 
> Legal & General Assurance (Pensions Management) Limited (no 1006112) is 
> authorised by the Prudential Regulation Authority and regulated by the 
> Financial Conduct Authority and the Prudential Regulation Authority. It is 
> registered in England & Wales with a registered office at One Coleman Street, 
> London, EC2R 5AA. 
> 
> Legal & General Property Limited (no 2091897) is authorised and regulated by 
> the Financial Conduct Authority for insurance mediation activities. It is 
> registered in England & Wales with a registered office at One Coleman Street, 
> London, EC2R 5AA. 
> 
> LGIM Managers (Europe) Limited is authorised and regulated by the Central 
> Bank of Ireland (C173733). It is registered in the Republic of Ireland (no 
> 609677) with a registered office at 33/34 Sir John Rogerson's Quay, Dublin 2, 
> D02 XK09. 
> 
> Legal & General Group PLC, Registered Office One Coleman Street, London, EC2R 
> 5AA. 
> 
> Registered in England no: 1417162 
> 
>  This email has come from the internet and has been scanned for all 
> viruses and potentially offensive content by Messagelabs on behalf of Legal & 
> General 

Re: Urgent: HDFS processors throwing OOM - Compressed class space exception

[Jorge Machado]

How big are the files that you are trying to store? How much memory did you 
configure nifi ? 

> On 22. Jul 2020, at 06:13, Mohit Jain  wrote:
> 
> Hi team,
> 
> I’ve been facing the issue while using any HDFS processor, e.g. - PutHDFS 
> throws the error - 
> Failed to write to HDFS due to compressed class space: 
> java.lang.OutOFMemoryError
> 
> Eventually the node gets disconnected.
> 
> Any help would be appreciated.
> 
> Regards,
> Mohit
> 

Re: Spark 3 pod template for the driver

[Jorge Machado]

Try to set spark.kubernetes.container.image

> On 26. Jun 2020, at 14:58, Michel Sumbul  wrote:
> 
> Hi guys,
> 
> I try to use Spark 3 on top of Kubernetes and to specify a pod template for 
> the driver.
> 
> Here is my pod manifest or the driver and when I do a spark-submit with the 
> option:
> --conf 
> spark.kubernetes.driver.podTemplateFile=/data/k8s/podtemplate_driver3.yaml
> 
> I got the error message that I need to specify an image, but it's the 
> manifest.
> Does my manifest file is wrong, How should it look like?
> 
> Thanks for your help,
> Michel
> 
> 
> The pod manifest:
> 
> apiVersion: v1
> kind: Pod
> metadata:
>   name: mySpark3App
>   labels:
> app: mySpark3App
> customlabel/app-id: "1"
> spec:
>   securityContext:
> runAsUser: 1000
>   volumes:
> - name: "test-volume"
>   emptyDir: {}
>   containers:
> - name: spark3driver
>   image: mydockerregistry.example.com/images/dev/spark3:latest 
> 
>   instances: 1
>   resources:
> requests:
>   cpu: "1000m"
>   memory: "512Mi"
> limits:
>   cpu: "1000m"
>   memory: "512Mi"
>   volumeMounts:
>- name: "test-volume"
>  mountPath: "/tmp"

Re: Using hadoop-cloud_2.12 jars

[Jorge Machado]

You can build it from source. 

Clone the spark git repo and run: ./build/mvn clean package -DskipTests 
-Phadoop-3.2 -Pkubernetes -Phadoop-cloud

Regards


> On 22. Jun 2020, at 11:00, Rahij Ramsharan  wrote:
> 
> Hello,
> 
> I am trying to use the new S3 committers 
> (https://spark.apache.org/docs/latest/cloud-integration.html#committing-work-into-cloud-storage-safely-and-fast
>  
> )
>  in spark 3.0.0. As per 
> https://spark.apache.org/docs/latest/cloud-integration.html#installation 
> , I 
> need to include "org.apache.spark:hadoop-cloud_2.12:3.0.0" in my classpath. 
> However, I am not able to locate where it is published - 
> https://mvnrepository.com/artifact/org.apache.spark/hadoop-cloud 
>  is a 404 
> and https://mvnrepository.com/artifact/org.apache.spark/spark-hadoop-cloud 
>  has 
> only jars from CDH/Cloudera etc (and none for spark 3.0.0).
> 
> Is this intentional or is there some bug in the spark publishing code?
> 
> Thanks
> Rahij

[jira] [Commented] (SPARK-31683) Make Prometheus output consistent with DropWizard 4.1 result

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-31683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128625#comment-17128625
 ] 

Jorge Machado commented on SPARK-31683:
---

It would be great if we could use the RPC backend from spark to be able to 
aggregate this into the driver only This way we only need to scrape the driver. 
I have made an implementation based on yours that registers it to consul. This 
way we can discover the yarn applications via consul for example... 

> Make Prometheus output consistent with DropWizard 4.1 result
> 
>
> Key: SPARK-31683
> URL: https://issues.apache.org/jira/browse/SPARK-31683
> Project: Spark
>  Issue Type: Sub-task
>  Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Dongjoon Hyun
>Assignee: Dongjoon Hyun
>Priority: Major
> Fix For: 3.0.0
>
>
> SPARK-29032 adds Prometheus support.
> After that, SPARK-29674 upgraded DropWizard for JDK9+ support and causes 
> difference in output labels and number of keys.
>  
> This issue aims to fix this inconsistency in Spark.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-31683) Make Prometheus output consistent with DropWizard 4.1 result

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-31683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126671#comment-17126671
 ] 

Jorge Machado commented on SPARK-31683:
---

[~dongjoon] this only exporters the metrics from the driver right?

> Make Prometheus output consistent with DropWizard 4.1 result
> 
>
> Key: SPARK-31683
> URL: https://issues.apache.org/jira/browse/SPARK-31683
> Project: Spark
>  Issue Type: Sub-task
>  Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Dongjoon Hyun
>Assignee: Dongjoon Hyun
>Priority: Major
> Fix For: 3.0.0
>
>
> SPARK-29032 adds Prometheus support.
> After that, SPARK-29674 upgraded DropWizard for JDK9+ support and causes 
> difference in output labels and number of keys.
>  
> This issue aims to fix this inconsistency in Spark.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Re: Arrow RecordBatches/Pandas Dataframes to (Arrow enabled) Spark Dataframe conversion in streaming fashion

[Jorge Machado]

Hey, from what I know you can try to Union them df.union(df2)

Not sure if this is what you need 

> On 25. May 2020, at 13:53, Tanveer Ahmad - EWI  wrote:
> 
> Hi all,
> 
> I need some help regarding Arrow RecordBatches/Pandas Dataframes to (Arrow 
> enabled) Spark Dataframe conversions.
> Here the example explains very well how to convert a single Pandas Dataframe 
> to Spark Dataframe [1]. 
> 
> But in my case, some external applications are generating Arrow RecordBatches 
> in my PySpark application in streaming fashion. Each time I receive an Arrow 
> RB, I want to transfer/append it to a Spark Dataframe. So is it possible to 
> create a Spark Dataframe initially from one Arrow RecordBatch and then start 
> appending many other in-coming Arrow RecordBatches to that Spark Dataframe 
> (like in streaming fashion)? Thanks!
> 
> I saw another example [2] in which all the Arrow RB are being converted to 
> Spark Dataframe but my case is little bit different than this.  
> 
> [1] https://spark.apache.org/docs/latest/sql-pyspark-pandas-with-arrow.html 
>  
> [2] https://gist.github.com/linar-jether/7dd61ed6fa89098ab9c58a1ab428b2b5 
> 
> 
> ---
> Regards,
> Tanveer Ahmad

Re: Nifi - how to achieve a concurrent development and CI/CD

[Jorge Machado]

I think we could improve Nifi by hooking up to GitHub somehow. So that we don’t 
need the registry… 

> On 14. May 2020, at 18:49, Ami Goldenberg  wrote:
> 
> Hi Eric,
> Would love to know, what kind of tests do you write/run for NiFi? We were 
> just researching this topic.
> 
> On Thu, May 14, 2020 at 6:38 PM Eric Secules  <mailto:esecu...@gmail.com>> wrote:
> Hi Michal,
> 
> I'm also using a single registry for development and production. It doesn't 
> help with collaborating on the same process group as there is way for it to 
> reconcile merge conflicts. Instead, the registry will earn you that you're 
> about to overwrite someone else's changes. Another pain of concurrent 
> development is there's no concept of a PR and no visual diff of your local 
> changes making review difficult. I've backed our registry up to git I've set 
> up a CI pipeline in Azure Devops which runs our tests every time a new 
> version of a process group is checked in. It's better than nothing but I'd 
> rather nifi implemented git flow. Developing as a team on nifi and nifi 
> registry is like your whole team developing and pushing directly to master.
> 
> 
> -Eric
> 
> On Thu., May 14, 2020, 7:02 a.m. Jorge Machado,  <mailto:jom...@me.com>> wrote:
> Hi, 
> 
> Managing xml is always hard I think. Last time I need to do something similar 
> we used https://nifi.apache.org/registry.html 
> <https://nifi.apache.org/registry.html>
> Works pretty well It was already 2 Years ago. Maybe now there is something 
> better 
> 
>> On 14. May 2020, at 15:57, Michal Slama (DHL IT Services), external 
>> mailto:michal.sl...@dhl.com>> wrote:
>> 
>> Hello,
>>  
>> may I ask you for recommendations for development and CI/CD in NiFi?
>> Pls let me describe our situation…I am a developer from DHL currently 
>> working on project including NiFi. It is part of our core and it is 
>> responsible for handling incoming data streams, data transormation and put 
>> it into various elastic search indexes (and queues).
>>  
>> Up to now development was quite straightforward as only one developer did 
>> it. But we have extended our team recently and now we face problems how to 
>> correctly maintain development for more developers working in parallel and 
>> then CI/CD of it as we have classical dev/test/uat/project env. structure.
>>  
>> Pls do you have any recommadation how to achive it? For now in general is 
>> enought.  Its good to mention that currently we work with NiFi version 
>> 1.8…tried to upgrade to 1.9. but some of components failed so the upgrade 
>> was postponed. But with new features in 1.10. and 1.11. we head to uprade to 
>> these versions.
>>  
>> Maybe if we can arrange a call it would be great!
>>  
>> With regards,
>> Michal Sláma
>>  
>> 
>> 
>> This message is from DHL Information Services (Europe) s.r.o. and may 
>> contain confidential business information. It is intended solely for the use 
>> of the individual to whom it is addressed. If you are not the intended 
>> recipient please contact the sender and delete this message and any 
>> attachment from your system. Unauthorized publication, use, dissemination, 
>> forwarding, printing or copying of this email and its attachments is 
>> strictly prohibited.
> 

Re: Nifi - how to achieve a concurrent development and CI/CD

[Jorge Machado]

Hi, 

Managing xml is always hard I think. Last time I need to do something similar 
we used https://nifi.apache.org/registry.html 

Works pretty well It was already 2 Years ago. Maybe now there is something 
better 

> On 14. May 2020, at 15:57, Michal Slama (DHL IT Services), external 
>  wrote:
> 
> Hello,
>  
> may I ask you for recommendations for development and CI/CD in NiFi?
> Pls let me describe our situation…I am a developer from DHL currently working 
> on project including NiFi. It is part of our core and it is responsible for 
> handling incoming data streams, data transormation and put it into various 
> elastic search indexes (and queues).
>  
> Up to now development was quite straightforward as only one developer did it. 
> But we have extended our team recently and now we face problems how to 
> correctly maintain development for more developers working in parallel and 
> then CI/CD of it as we have classical dev/test/uat/project env. structure.
>  
> Pls do you have any recommadation how to achive it? For now in general is 
> enought.  Its good to mention that currently we work with NiFi version 
> 1.8…tried to upgrade to 1.9. but some of components failed so the upgrade was 
> postponed. But with new features in 1.10. and 1.11. we head to uprade to 
> these versions.
>  
> Maybe if we can arrange a call it would be great!
>  
> With regards,
> Michal Sláma
>  
> 
> 
> This message is from DHL Information Services (Europe) s.r.o. and may contain 
> confidential business information. It is intended solely for the use of the 
> individual to whom it is addressed. If you are not the intended recipient 
> please contact the sender and delete this message and any attachment from 
> your system. Unauthorized publication, use, dissemination, forwarding, 
> printing or copying of this email and its attachments is strictly prohibited.

Re: How to deal Schema Evolution with Dataset API

[Jorge Machado]

Ok, I found a way to solve it. 

Just pass the schema like this: 

val schema = Encoders.product[Person].schema

spark.read.schema(schema).parquet(“input”)….

> On 9. May 2020, at 13:28, Jorge Machado  wrote:
> 
> Hello everyone, 
> 
> One question to the community. 
> 
> Imagine I have this
> 
>   Case class Person(age: int)
> 
>   spark.read.parquet(“inputPath”).as[Person]
> 
> 
> After a few weeks of coding I change the class to: 
>   Case class Person(age: int, name: Option[String] = None)
> 
> 
> Then when I run the new code on the same input it fails saying that It cannot 
> find the name on the schema from the parquet file. 
> 
> Spark version 2.3.3
> 
> How is the best way to guard or fix this? Regenerating all data seems not to 
> be a option for us. 
> 
> Thx
> -
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
> 


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

How to deal Schema Evolution with Dataset API

[Jorge Machado]

Hello everyone, 

One question to the community. 

Imagine I have this

Case class Person(age: int)

spark.read.parquet(“inputPath”).as[Person]


After a few weeks of coding I change the class to: 
Case class Person(age: int, name: Option[String] = None)


Then when I run the new code on the same input it fails saying that It cannot 
find the name on the schema from the parquet file. 

Spark version 2.3.3

How is the best way to guard or fix this? Regenerating all data seems not to be 
a option for us. 

Thx
-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

How to deal Schema Evolution with Dataset API

[Jorge Machado]

Hello everyone, 

One question to the community. 

Imagine I have this

Case class Person(age: int)

spark.read.parquet(“inputPath”).as[Person]


After a few weeks of coding I change the class to: 
Case class Person(age: int, name: Option[String] = None)


Then when I run the new code on the same input it fails saying that It cannot 
find the name on the schema from the parquet file. 

Spark version 2.3.3

How is the best way to guard or fix this? Regenerating all data seems not to be 
a option for us. 

Thx 

[jira] [Commented] (SPARK-26902) Support java.time.Instant as an external type of TimestampType

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-26902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17085759#comment-17085759
 ] 

Jorge Machado commented on SPARK-26902:
---

what about Supporting the interface Temporal ?

> Support java.time.Instant as an external type of TimestampType
> --
>
> Key: SPARK-26902
> URL: https://issues.apache.org/jira/browse/SPARK-26902
> Project: Spark
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 2.4.0
>Reporter: Maxim Gekk
>Assignee: Maxim Gekk
>Priority: Major
> Fix For: 3.0.0
>
>
> Currently, Spark supports the java.sql.Date and java.sql.Timestamp types as 
> external types for Catalyst's DateType and TimestampType. It accepts and 
> produces values of such types. Since Java 8, base classes for dates and 
> timestamps are java.time.Instant, java.time.LocalDate/LocalDateTime, and 
> java.time.ZonedDateTime. Need to add new converters from/to Instant.
> The Instant type holds epoch seconds (and nanoseconds), and directly reflects 
> to Catalyst's TimestampType.
> Main motivations for the changes:
> - Smoothly support Java 8 time API
> - Avoid inconsistency of calendars used inside Spark 3.0 (Proleptic Gregorian 
> calendar) and inside of java.sql.Timestamp (hybrid calendar - Julian + 
> Gregorian). 
> - Make conversion independent from current system timezone.
> In case of collecting values of Date/TimestampType, the following SQL config 
> can control types of returned values:
>  - spark.sql.catalyst.timestampType with supported values 
> "java.sql.Timestamp" (by default) and "java.time.Instant"



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-30272) Remove usage of Guava that breaks in Guava 27

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-30272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070774#comment-17070774
 ] 

Jorge Machado commented on SPARK-30272:
---

I failed to fix the guava stuff of course ... Today morning I tried to 
replicate the problem of the missing azure-hadoop jar but It seems to be 
working without any patch from my side. . I assume that I did something wrong 
on build.  Just for reference my steps: 
{code:java}
git checkout v3.0.0-preview-rc2
./build/mvn clean package -DskipTests -Phadoop-3.2 -Pkubernetes 
-Phadoop-cloud
./bin/docker-image-tool.sh -r docker.io/myrepo -t v2.3.0 -p 
kubernetes/dockerfiles/spark/bindings/python/Dockerfile build
 docker run --rm -it myrepo/spark:v2.3.0 bash
 185@57fb3dd68902:/opt/spark/jars$ ls -altr *azure*
-rw-r--r-- 1 root root  67314 Mar 28 17:15 
hadoop-azure-datalake-3.2.0.jar
-rw-r--r-- 1 root root 480512 Mar 28 17:15 hadoop-azure-3.2.0.jar
-rw-r--r-- 1 root root 812977 Mar 28 17:15 azure-storage-7.0.0.jar
-rw-r--r-- 1 root root  10288 Mar 28 17:15 azure-keyvault-core-1.0.0.jar
-rw-r--r-- 1 root root  94061 Mar 28 17:15 
azure-data-lake-store-sdk-2.2.9.jar
{code}
As you see the hadoop-azure is there but not on version 3.2.1 but I guess this 
is a matter of updating the pom. 

 

> Remove usage of Guava that breaks in Guava 27
> -
>
> Key: SPARK-30272
> URL: https://issues.apache.org/jira/browse/SPARK-30272
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core, SQL
>Affects Versions: 3.0.0
>Reporter: Sean R. Owen
>Assignee: Sean R. Owen
>Priority: Major
> Fix For: 3.0.0
>
>
> Background:
> https://issues.apache.org/jira/browse/SPARK-29250
> https://github.com/apache/spark/pull/25932
> Hadoop 3.2.1 will update Guava from 11 to 27. There are a number of methods 
> that changed between those releases, typically just a rename, but, means one 
> set of code can't work with both, while we want to work with Hadoop 2.x and 
> 3.x. Among them:
> - Objects.toStringHelper was moved to MoreObjects; we can just use the 
> Commons Lang3 equivalent
> - Objects.hashCode etc were renamed; use java.util.Objects equivalents
> - MoreExecutors.sameThreadExecutor() became directExecutor(); for same-thread 
> execution we can use a dummy implementation of ExecutorService / Executor
> - TypeToken.isAssignableFrom become isSupertypeOf; work around with reflection
> There is probably more to the Guava issue than just this change, but it will 
> make Spark itself work with more versions and reduce our exposure to Guava 
> along the way anyway.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (HADOOP-15669) ABFS: Improve HTTPS Performance

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070519#comment-17070519
 ] 

Jorge Machado commented on HADOOP-15669:


I‘m using the docker images that the docker-image-tool.sh provides ... 

> ABFS: Improve HTTPS Performance
> ---
>
> Key: HADOOP-15669
> URL: https://issues.apache.org/jira/browse/HADOOP-15669
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Reporter: Thomas Marqardt
>Assignee: Vishwajeet Dusane
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: ABFS - Improve HTTPS Performance Over Java Based 
> Client.pdf, HADOOP-15669-HADOOP-15407-01.patch, 
> HADOOP-15669-HADOOP-15407-02.patch, HADOOP-15669-HADOOP-15407-03.patch, 
> HADOOP-15669-HADOOP-15407-04.patch
>
>
> We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP.  Lets 
> perform a detailed measurement and see what can be done to improve throughput.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Commented] (SPARK-30272) Remove usage of Guava that breaks in Guava 27

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-30272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070255#comment-17070255
 ] 

Jorge Machado commented on SPARK-30272:
---

So I was able to fix it. I build it with profile hadoop 3.2 but after the build 
the hadoop-azure.jar is missing so I added manually into my container and now 
it seems to load. 

I was trying to put guava 28 and remove the 14 but this is a lot of work... why 
do we use a old guava version ? 

> Remove usage of Guava that breaks in Guava 27
> -
>
> Key: SPARK-30272
> URL: https://issues.apache.org/jira/browse/SPARK-30272
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core, SQL
>Affects Versions: 3.0.0
>Reporter: Sean R. Owen
>Assignee: Sean R. Owen
>Priority: Major
> Fix For: 3.0.0
>
>
> Background:
> https://issues.apache.org/jira/browse/SPARK-29250
> https://github.com/apache/spark/pull/25932
> Hadoop 3.2.1 will update Guava from 11 to 27. There are a number of methods 
> that changed between those releases, typically just a rename, but, means one 
> set of code can't work with both, while we want to work with Hadoop 2.x and 
> 3.x. Among them:
> - Objects.toStringHelper was moved to MoreObjects; we can just use the 
> Commons Lang3 equivalent
> - Objects.hashCode etc were renamed; use java.util.Objects equivalents
> - MoreExecutors.sameThreadExecutor() became directExecutor(); for same-thread 
> execution we can use a dummy implementation of ExecutorService / Executor
> - TypeToken.isAssignableFrom become isSupertypeOf; work around with reflection
> There is probably more to the Guava issue than just this change, but it will 
> make Spark itself work with more versions and reduce our exposure to Guava 
> along the way anyway.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (HADOOP-15669) ABFS: Improve HTTPS Performance

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/HADOOP-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17070223#comment-17070223
 ] 

Jorge Machado commented on HADOOP-15669:


Hi all, 

I'm still seeing this "20/03/28 19:58:22 WARN SSLSocketFactoryEx: Failed to 
load OpenSSL. Falling back to the JSSE default." On the logs Any tipps ? 
Using hadoop-azure 3.2.0

> ABFS: Improve HTTPS Performance
> ---
>
> Key: HADOOP-15669
> URL: https://issues.apache.org/jira/browse/HADOOP-15669
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs/azure
>Reporter: Thomas Marqardt
>Assignee: Vishwajeet Dusane
>Priority: Major
> Fix For: 3.2.0
>
> Attachments: ABFS - Improve HTTPS Performance Over Java Based 
> Client.pdf, HADOOP-15669-HADOOP-15407-01.patch, 
> HADOOP-15669-HADOOP-15407-02.patch, HADOOP-15669-HADOOP-15407-03.patch, 
> HADOOP-15669-HADOOP-15407-04.patch
>
>
> We see approximately 50% worse throughput for ABFS over HTTPs vs HTTP.  Lets 
> perform a detailed measurement and see what can be done to improve throughput.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (SPARK-30272) Remove usage of Guava that breaks in Guava 27

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-30272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069933#comment-17069933
 ] 

Jorge Machado edited comment on SPARK-30272 at 3/28/20, 4:25 PM:
-

Hey Sean, 

This seems still to make problems for example: 
{code:java}
> $ ./bin/run-example SparkPi 100                                               
>                                                                               
>                                                                               
>     [±master ●]> $ ./bin/run-example SparkPi 100                              
>                                                                               
>                                                                               
>                      [±master ●]20/03/28 17:21:13 WARN Utils: Your hostname, 
> Jorges-MacBook-Pro.local resolves to a loopback address: 127.0.0.1; using 
> 192.168.1.2 instead (on interface en0)20/03/28 17:21:13 WARN Utils: Set 
> SPARK_LOCAL_IP if you need to bind to another address20/03/28 17:21:14 WARN 
> NativeCodeLoader: Unable to load native-hadoop library for your platform... 
> using builtin-java classes where applicableUsing Spark's default log4j 
> profile: org/apache/spark/log4j-defaults.properties20/03/28 17:21:14 INFO 
> SparkContext: Running Spark version 3.1.0-SNAPSHOT20/03/28 17:21:14 INFO 
> ResourceUtils: 
> ==20/03/28 
> 17:21:14 INFO ResourceUtils: No custom resources configured for 
> spark.driver.20/03/28 17:21:14 INFO ResourceUtils: 
> ==20/03/28 
> 17:21:14 INFO SparkContext: Submitted application: Spark Pi20/03/28 17:21:14 
> INFO ResourceProfile: Default ResourceProfile created, executor resources: 
> Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: 
> memory, amount: 1024, script: , vendor: ), task resources: Map(cpus -> name: 
> cpus, amount: 1.0)20/03/28 17:21:14 INFO ResourceProfile: Limiting resource 
> is cpu20/03/28 17:21:14 INFO ResourceProfileManager: Added ResourceProfile 
> id: 020/03/28 17:21:14 INFO SecurityManager: Changing view acls to: 
> jorge20/03/28 17:21:14 INFO SecurityManager: Changing modify acls to: 
> jorge20/03/28 17:21:14 INFO SecurityManager: Changing view acls groups 
> to:20/03/28 17:21:14 INFO SecurityManager: Changing modify acls groups 
> to:20/03/28 17:21:14 INFO SecurityManager: SecurityManager: authentication 
> disabled; ui acls disabled; users  with view permissions: Set(jorge); groups 
> with view permissions: Set(); users  with modify permissions: Set(jorge); 
> groups with modify permissions: Set()20/03/28 17:21:14 INFO Utils: 
> Successfully started service 'sparkDriver' on port 58192.20/03/28 17:21:14 
> INFO SparkEnv: Registering MapOutputTracker20/03/28 17:21:14 INFO SparkEnv: 
> Registering BlockManagerMaster20/03/28 17:21:14 INFO 
> BlockManagerMasterEndpoint: Using 
> org.apache.spark.storage.DefaultTopologyMapper for getting topology 
> information20/03/28 17:21:14 INFO BlockManagerMasterEndpoint: 
> BlockManagerMasterEndpoint up20/03/28 17:21:14 INFO SparkEnv: Registering 
> BlockManagerMasterHeartbeat20/03/28 17:21:14 INFO DiskBlockManager: Created 
> local directory at 
> /private/var/folders/0h/5b7dw9p11l58hyk0_s0d3cnhgn/T/blockmgr-d9e88815-075e-4c9b-9cc8-21c72e97c86920/03/28
>  17:21:14 INFO MemoryStore: MemoryStore started with capacity 366.3 
> MiB20/03/28 17:21:14 INFO SparkEnv: Registering 
> OutputCommitCoordinator20/03/28 17:21:15 INFO Utils: Successfully started 
> service 'SparkUI' on port 4040.20/03/28 17:21:15 INFO SparkUI: Bound SparkUI 
> to 0.0.0.0, and started at http://192.168.1.2:404020/03/28 17:21:15 INFO 
> SparkContext: Added JAR 
> file:///Users/jorge/Downloads/spark/dist/examples/jars/spark-examples_2.12-3.1.0-SNAPSHOT.jar
>  at spark://192.168.1.2:58192/jars/spark-examples_2.12-3.1.0-SNAPSHOT.jar 
> with timestamp 158541247516620/03/28 17:21:15 INFO SparkContext: Added JAR 
> file:///Users/jorge/Downloads/spark/dist/examples/jars/scopt_2.12-3.7.1.jar 
> at spark://192.168.1.2:58192/jars/scopt_2.12-3.7.1.jar with timestamp 
> 158541247516620/03/28 17:21:15 INFO Executor: Starting executor ID driver on 
> host 192.168.1.220/03/28 17:21:15 INFO Utils: Successfully started service 
> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 
> 58193.20/03/28 17:21:15 INFO NettyBlockTransferService: Server created on 
> 192.168.1.2:5819320/03/28 17:21:15 INFO BlockManager: Using 
> org.apache.spark.storage.RandomBlockReplicationPolicy for block replication 
> policyException in thread "main" java.lang.NoClassDefFo

[jira] [Commented] (SPARK-30272) Remove usage of Guava that breaks in Guava 27

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-30272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069933#comment-17069933
 ] 

Jorge Machado commented on SPARK-30272:
---

Hey Sean, 

This seems still to make problems for example: 
{code:java}
 java.lang.NoClassDefFoundError: 
com/google/common/util/concurrent/internal/InternalFutureFailureAccess 
java.lang.NoClassDefFoundError: 
com/google/common/util/concurrent/internal/InternalFutureFailureAccess at 
java.lang.ClassLoader.defineClass1(Native Method) at 
java.lang.ClassLoader.defineClass(ClassLoader.java:757) at 
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at 
java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at 
java.net.URLClassLoader.access$100(URLClassLoader.java:74) at 
java.net.URLClassLoader$1.run(URLClassLoader.java:369) at 
java.net.URLClassLoader$1.run(URLClassLoader.java:363) at 
java.security.AccessController.doPrivileged(Native Method) at 
java.net.URLClassLoader.findClass(URLClassLoader.java:362) at 
java.lang.ClassLoader.loadClass(ClassLoader.java:419) at 
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at 
java.lang.ClassLoader.loadClass(ClassLoader.java:352) at 
java.lang.ClassLoader.defineClass1(Native Method) at 
java.lang.ClassLoader.defineClass(ClassLoader.java:757) at 
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at 
java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at 
java.net.URLClassLoader.access$100(URLClassLoader.java:74) at 
java.net.URLClassLoader$1.run(URLClassLoader.java:369) at 
java.net.URLClassLoader$1.run(URLClassLoader.java:363) at 
java.security.AccessController.doPrivileged(Native Method) at 
java.net.URLClassLoader.findClass(URLClassLoader.java:362) at 
java.lang.ClassLoader.loadClass(ClassLoader.java:419) at 
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at 
java.lang.ClassLoader.loadClass(ClassLoader.java:352) at 
java.lang.ClassLoader.defineClass1(Native Method) at 
java.lang.ClassLoader.defineClass(ClassLoader.java:757) at 
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at 
java.net.URLClassLoader.defineClass(URLClassLoader.java:468) at 
java.net.URLClassLoader.access$100(URLClassLoader.java:74) at 
java.net.URLClassLoader$1.run(URLClassLoader.java:369) at 
java.net.URLClassLoader$1.run(URLClassLoader.java:363) at 
java.security.AccessController.doPrivileged(Native Method) at 
java.net.URLClassLoader.findClass(URLClassLoader.java:362) at 
java.lang.ClassLoader.loadClass(ClassLoader.java:419) at 
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at 
java.lang.ClassLoader.loadClass(ClassLoader.java:352) at 
com.google.common.cache.LocalCache$LoadingValueReference.(LocalCache.java:3472)
 at 
com.google.common.cache.LocalCache$LoadingValueReference.(LocalCache.java:3476)
 at 
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2134)
 at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2045) at 
com.google.common.cache.LocalCache.get(LocalCache.java:3951) at 
com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974) at 
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958) 
at org.apache.hadoop.security.Groups.getGroups(Groups.java:228) at 
org.apache.hadoop.security.UserGroupInformation.getGroups(UserGroupInformation.java:1588)
 at 
org.apache.hadoop.security.UserGroupInformation.getPrimaryGroupName(UserGroupInformation.java:1453)
 at 
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.(AzureBlobFileSystemStore.java:147)
 at 
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:104)
 at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3303) at 
org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:124) at 
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3352) at 
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3320) at 
org.apache.hadoop.fs.FileSystem.get(FileSystem.java:479) at 
org.apache.hadoop.fs.Path.getFileSystem(Path.java:365) at 
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInputFormat.java:522)
 at 
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(FileInputFormat.java:491)
 at 
org.apache.spark.SparkContext.$anonfun$newAPIHadoopFile$2(SparkContext.scala:1219)
 at 
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) 
at 
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) 
at org.apache.spark.SparkContext.withScope(SparkContext.scala:757) at 
org.apache.spark.SparkContext.newAPIHadoopFile(SparkContext.scala:1207) at 
org.apache.spark.api.java.JavaSparkContext.newAPIHadoopFile(JavaSparkContext.scala:484)
{code}
I still see a lot of references to guava 14 on master is this normal ? Sorry 
for the question...

 

 

 

> Remove usage of Gu

[jira] [Comment Edited] (SPARK-23897) Guava version

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-23897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069352#comment-17069352
 ] 

Jorge Machado edited comment on SPARK-23897 at 3/28/20, 9:50 AM:
-

I think that master is actually broken at least for commit 
d025ddbaa7e7b9746d8e47aeed61ed39d2f09f0e. I builded with: 
{code:java}
./build/mvn  clean package  -DskipTests -Phadoop-3.2 -Pkubernetes -Phadoop-cloud
{code}
I get: 
{code:java}
jorge@Jorges-MacBook-Pro ~/Downloads/spark/dist/bin                             
                                                                                
                                                                                
   [10:43:24]jorge@Jorges-MacBook-Pro ~/Downloads/spark/dist/bin                
                                                                                
                                                                                
                [10:43:24]> $ java -version                                     
                                                                                
                                                                                
                            [±master ✓]java version "1.8.0_211"Java(TM) SE 
Runtime Environment (build 1.8.0_211-b12)Java HotSpot(TM) 64-Bit Server VM 
(build 25.211-b12, mixed mode) jorge@Jorges-MacBook-Pro 
~/Downloads/spark/dist/bin                                                      
                                                                                
                                                          [10:43:27]> $ 
./run-example SparkPi 100                                                       
                                                                                
                                                                              
[±master ✓]Exception in thread "main" java.lang.NoSuchMethodError: 
com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
 at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357) at 
org.apache.hadoop.conf.Configuration.set(Configuration.java:1338) at 
org.apache.spark.deploy.SparkHadoopUtil$.org$apache$spark$deploy$SparkHadoopUtil$$appendS3AndSparkHadoopHiveConfigurations(SparkHadoopUtil.scala:456)
 at 
org.apache.spark.deploy.SparkHadoopUtil$.newConfiguration(SparkHadoopUtil.scala:427)
 at 
org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$2(SparkSubmit.scala:342)
 at scala.Option.getOrElse(Option.scala:189) at 
org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:342)
 at 
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
 at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at 
org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at 
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at 
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007) at 
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016) at 
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala){code}
If I delete guava 14 and add guava28 it works.


was (Author: jomach):
I think that master is actually broken at least for commit 
d025ddbaa7e7b9746d8e47aeed61ed39d2f09f0e. I builded with: 
{code:java}
./build/mvn  clean package  -DskipTests -Phadoop-3.2 -Pkubernetes -Phadoop-cloud
{code}
I get: 
{code:java}
jorge@Jorges-MacBook-Pro ~/Downloads/spark/dist/bin                             
                                                                                
                                                                                
   [10:43:24]jorge@Jorges-MacBook-Pro ~/Downloads/spark/dist/bin                
                                                                                
                                                                                
                [10:43:24]> $ java -version                                     
                                                                                
                                                                                
                            [±master ✓]java version "1.8.0_211"Java(TM) SE 
Runtime Environment (build 1.8.0_211-b12)Java HotSpot(TM) 64-Bit Server VM 
(build 25.211-b12, mixed mode)
jorge@Jorges-MacBook-Pro ~/Downloads/spark/dist/bin                             
                                                                                
                                                                                
   [10:43:27]> $ ./run-example SparkPi 100                                      
                                                                                
                                                              

[jira] [Commented] (SPARK-23897) Guava version

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-23897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17069352#comment-17069352
 ] 

Jorge Machado commented on SPARK-23897:
---

I think that master is actually broken at least for commit 
d025ddbaa7e7b9746d8e47aeed61ed39d2f09f0e. I builded with: 
{code:java}
./build/mvn  clean package  -DskipTests -Phadoop-3.2 -Pkubernetes -Phadoop-cloud
{code}
I get: 
{code:java}
jorge@Jorges-MacBook-Pro ~/Downloads/spark/dist/bin                             
                                                                                
                                                                                
   [10:43:24]jorge@Jorges-MacBook-Pro ~/Downloads/spark/dist/bin                
                                                                                
                                                                                
                [10:43:24]> $ java -version                                     
                                                                                
                                                                                
                            [±master ✓]java version "1.8.0_211"Java(TM) SE 
Runtime Environment (build 1.8.0_211-b12)Java HotSpot(TM) 64-Bit Server VM 
(build 25.211-b12, mixed mode)
jorge@Jorges-MacBook-Pro ~/Downloads/spark/dist/bin                             
                                                                                
                                                                                
   [10:43:27]> $ ./run-example SparkPi 100                                      
                                                                                
                                                                                
               [±master ✓]Exception in thread "main" 
java.lang.NoSuchMethodError: 
com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
 at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357) at 
org.apache.hadoop.conf.Configuration.set(Configuration.java:1338) at 
org.apache.spark.deploy.SparkHadoopUtil$.org$apache$spark$deploy$SparkHadoopUtil$$appendS3AndSparkHadoopHiveConfigurations(SparkHadoopUtil.scala:456)
 at 
org.apache.spark.deploy.SparkHadoopUtil$.newConfiguration(SparkHadoopUtil.scala:427)
 at 
org.apache.spark.deploy.SparkSubmit.$anonfun$prepareSubmitEnvironment$2(SparkSubmit.scala:342)
 at scala.Option.getOrElse(Option.scala:189) at 
org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:342)
 at 
org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:871)
 at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180) at 
org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203) at 
org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90) at 
org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1007) at 
org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1016) at 
org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
{code}

> Guava version
> -
>
> Key: SPARK-23897
> URL: https://issues.apache.org/jira/browse/SPARK-23897
> Project: Spark
>  Issue Type: Dependency upgrade
>  Components: Spark Core
>Affects Versions: 2.3.0
>Reporter: Sercan Karaoglu
>Priority: Minor
>
> Guava dependency version 14 is pretty old, needs to be updated to at least 
> 16, google cloud storage connector uses newer one which causes pretty popular 
> error with guava; "java.lang.NoSuchMethodError: 
> com.google.common.base.Splitter.splitToList(Ljava/lang/CharSequence;)Ljava/util/List;"
>  and causes app to crash



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-26412) Allow Pandas UDF to take an iterator of pd.DataFrames

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-26412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17046958#comment-17046958
 ] 

Jorge Machado commented on SPARK-26412:
---

Thanks for the Tipp. It helps 

> Allow Pandas UDF to take an iterator of pd.DataFrames
> -
>
> Key: SPARK-26412
> URL: https://issues.apache.org/jira/browse/SPARK-26412
> Project: Spark
>  Issue Type: New Feature
>  Components: PySpark
>Affects Versions: 3.0.0
>Reporter: Xiangrui Meng
>Assignee: Weichen Xu
>Priority: Major
> Fix For: 3.0.0
>
>
> Pandas UDF is the ideal connection between PySpark and DL model inference 
> workload. However, user needs to load the model file first to make 
> predictions. It is common to see models of size ~100MB or bigger. If the 
> Pandas UDF execution is limited to each batch, user needs to repeatedly load 
> the same model for every batch in the same python worker process, which is 
> inefficient.
> We can provide users the iterator of batches in pd.DataFrame and let user 
> code handle it:
> {code}
> @pandas_udf(DoubleType(), PandasUDFType.SCALAR_ITER)
> def predict(batch_iter):
>   model = ... # load model
>   for batch in batch_iter:
> yield model.predict(batch)
> {code}
> The type of each batch is:
> * a pd.Series if UDF is called with a single non-struct-type column
> * a tuple of pd.Series if UDF is called with more than one Spark DF columns
> * a pd.DataFrame if UDF is called with a single StructType column
> Examples:
> {code}
> @pandas_udf(...)
> def evaluate(batch_iter):
>   model = ... # load model
>   for features, label in batch_iter:
> pred = model.predict(features)
> yield (pred - label).abs()
> df.select(evaluate(col("features"), col("label")).alias("err"))
> {code}
> {code}
> @pandas_udf(...)
> def evaluate(pdf_iter):
>   model = ... # load model
>   for pdf in pdf_iter:
> pred = model.predict(pdf['x'])
> yield (pred - pdf['y']).abs()
> df.select(evaluate(struct(col("features"), col("label"))).alias("err"))
> {code}
> If the UDF doesn't return the same number of records for the entire 
> partition, user should see an error. We don't restrict that every yield 
> should match the input batch size.
> Another benefit is with iterator interface and asyncio from Python, it is 
> flexible for users to implement data pipelining.
> cc: [~icexelloss] [~bryanc] [~holdenk] [~hyukjin.kwon] [~ueshin] [~smilegator]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-26412) Allow Pandas UDF to take an iterator of pd.DataFrames

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-26412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17046373#comment-17046373
 ] 

Jorge Machado commented on SPARK-26412:
---

Well I was thinking on something more. like I would like to give a,b,c to 
another object. Like

 
{code:java}
class SomeClass():
  def __init__(a,b,c):
 pass

def map_func(batch_iter):

   dataset = SomeClass(batch_iter[0], batch_iter[1], batch_iter[2]) <- this 
does not work. 

{code}
and another thing, it would be great if we could just yield a json for example 
instead of this fixed types

> Allow Pandas UDF to take an iterator of pd.DataFrames
> -
>
> Key: SPARK-26412
> URL: https://issues.apache.org/jira/browse/SPARK-26412
> Project: Spark
>  Issue Type: New Feature
>  Components: PySpark
>Affects Versions: 3.0.0
>Reporter: Xiangrui Meng
>Assignee: Weichen Xu
>Priority: Major
> Fix For: 3.0.0
>
>
> Pandas UDF is the ideal connection between PySpark and DL model inference 
> workload. However, user needs to load the model file first to make 
> predictions. It is common to see models of size ~100MB or bigger. If the 
> Pandas UDF execution is limited to each batch, user needs to repeatedly load 
> the same model for every batch in the same python worker process, which is 
> inefficient.
> We can provide users the iterator of batches in pd.DataFrame and let user 
> code handle it:
> {code}
> @pandas_udf(DoubleType(), PandasUDFType.SCALAR_ITER)
> def predict(batch_iter):
>   model = ... # load model
>   for batch in batch_iter:
> yield model.predict(batch)
> {code}
> The type of each batch is:
> * a pd.Series if UDF is called with a single non-struct-type column
> * a tuple of pd.Series if UDF is called with more than one Spark DF columns
> * a pd.DataFrame if UDF is called with a single StructType column
> Examples:
> {code}
> @pandas_udf(...)
> def evaluate(batch_iter):
>   model = ... # load model
>   for features, label in batch_iter:
> pred = model.predict(features)
> yield (pred - label).abs()
> df.select(evaluate(col("features"), col("label")).alias("err"))
> {code}
> {code}
> @pandas_udf(...)
> def evaluate(pdf_iter):
>   model = ... # load model
>   for pdf in pdf_iter:
> pred = model.predict(pdf['x'])
> yield (pred - pdf['y']).abs()
> df.select(evaluate(struct(col("features"), col("label"))).alias("err"))
> {code}
> If the UDF doesn't return the same number of records for the entire 
> partition, user should see an error. We don't restrict that every yield 
> should match the input batch size.
> Another benefit is with iterator interface and asyncio from Python, it is 
> flexible for users to implement data pipelining.
> cc: [~icexelloss] [~bryanc] [~holdenk] [~hyukjin.kwon] [~ueshin] [~smilegator]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-26412) Allow Pandas UDF to take an iterator of pd.DataFrames

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-26412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17046265#comment-17046265
 ] 

Jorge Machado commented on SPARK-26412:
---

Hi, one question. 

when using "a tuple of pd.Series if UDF is called with more than one Spark DF 
columns" how can I get the Series into a variables. like 

a, b, c = iterator ? map seems not to be a python tuple ... 

> Allow Pandas UDF to take an iterator of pd.DataFrames
> -
>
> Key: SPARK-26412
> URL: https://issues.apache.org/jira/browse/SPARK-26412
> Project: Spark
>  Issue Type: New Feature
>  Components: PySpark
>Affects Versions: 3.0.0
>Reporter: Xiangrui Meng
>Assignee: Weichen Xu
>Priority: Major
> Fix For: 3.0.0
>
>
> Pandas UDF is the ideal connection between PySpark and DL model inference 
> workload. However, user needs to load the model file first to make 
> predictions. It is common to see models of size ~100MB or bigger. If the 
> Pandas UDF execution is limited to each batch, user needs to repeatedly load 
> the same model for every batch in the same python worker process, which is 
> inefficient.
> We can provide users the iterator of batches in pd.DataFrame and let user 
> code handle it:
> {code}
> @pandas_udf(DoubleType(), PandasUDFType.SCALAR_ITER)
> def predict(batch_iter):
>   model = ... # load model
>   for batch in batch_iter:
> yield model.predict(batch)
> {code}
> The type of each batch is:
> * a pd.Series if UDF is called with a single non-struct-type column
> * a tuple of pd.Series if UDF is called with more than one Spark DF columns
> * a pd.DataFrame if UDF is called with a single StructType column
> Examples:
> {code}
> @pandas_udf(...)
> def evaluate(batch_iter):
>   model = ... # load model
>   for features, label in batch_iter:
> pred = model.predict(features)
> yield (pred - label).abs()
> df.select(evaluate(col("features"), col("label")).alias("err"))
> {code}
> {code}
> @pandas_udf(...)
> def evaluate(pdf_iter):
>   model = ... # load model
>   for pdf in pdf_iter:
> pred = model.predict(pdf['x'])
> yield (pred - pdf['y']).abs()
> df.select(evaluate(struct(col("features"), col("label"))).alias("err"))
> {code}
> If the UDF doesn't return the same number of records for the entire 
> partition, user should see an error. We don't restrict that every yield 
> should match the input batch size.
> Another benefit is with iterator interface and asyncio from Python, it is 
> flexible for users to implement data pipelining.
> cc: [~icexelloss] [~bryanc] [~holdenk] [~hyukjin.kwon] [~ueshin] [~smilegator]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-24615) SPIP: Accelerator-aware task scheduling for Spark

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034492#comment-17034492
 ] 

Jorge Machado commented on SPARK-24615:
---

Yeah, that was my question. Thanks for the response. I will look at rapid.ai 
and try to use it inside a partition or so... 

> SPIP: Accelerator-aware task scheduling for Spark
> -
>
> Key: SPARK-24615
> URL: https://issues.apache.org/jira/browse/SPARK-24615
> Project: Spark
>  Issue Type: Epic
>  Components: Spark Core
>Affects Versions: 2.4.0
>Reporter: Saisai Shao
>Assignee: Thomas Graves
>Priority: Major
>  Labels: Hydrogen, SPIP
> Attachments: Accelerator-aware scheduling in Apache Spark 3.0.pdf, 
> SPIP_ Accelerator-aware scheduling.pdf
>
>
> (The JIRA received a major update on 2019/02/28. Some comments were based on 
> an earlier version. Please ignore them. New comments start at 
> [#comment-16778026].)
> h2. Background and Motivation
> GPUs and other accelerators have been widely used for accelerating special 
> workloads, e.g., deep learning and signal processing. While users from the AI 
> community use GPUs heavily, they often need Apache Spark to load and process 
> large datasets and to handle complex data scenarios like streaming. YARN and 
> Kubernetes already support GPUs in their recent releases. Although Spark 
> supports those two cluster managers, Spark itself is not aware of GPUs 
> exposed by them and hence Spark cannot properly request GPUs and schedule 
> them for users. This leaves a critical gap to unify big data and AI workloads 
> and make life simpler for end users.
> To make Spark be aware of GPUs, we shall make two major changes at high level:
> * At cluster manager level, we update or upgrade cluster managers to include 
> GPU support. Then we expose user interfaces for Spark to request GPUs from 
> them.
> * Within Spark, we update its scheduler to understand available GPUs 
> allocated to executors, user task requests, and assign GPUs to tasks properly.
> Based on the work done in YARN and Kubernetes to support GPUs and some 
> offline prototypes, we could have necessary features implemented in the next 
> major release of Spark. You can find a detailed scoping doc here, where we 
> listed user stories and their priorities.
> h2. Goals
> * Make Spark 3.0 GPU-aware in standalone, YARN, and Kubernetes.
> * No regression on scheduler performance for normal jobs.
> h2. Non-goals
> * Fine-grained scheduling within one GPU card.
> ** We treat one GPU card and its memory together as a non-divisible unit.
> * Support TPU.
> * Support Mesos.
> * Support Windows.
> h2. Target Personas
> * Admins who need to configure clusters to run Spark with GPU nodes.
> * Data scientists who need to build DL applications on Spark.
> * Developers who need to integrate DL features on Spark.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-24615) SPIP: Accelerator-aware task scheduling for Spark

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17034277#comment-17034277
 ] 

Jorge Machado commented on SPARK-24615:
---

[~tgraves] thanks for the input. It would be great to have one or two examples 
on how to use the GPUs within a dataset. 

I tried to figure out the api but I did not find any useful docs. Any tip?

> SPIP: Accelerator-aware task scheduling for Spark
> -
>
> Key: SPARK-24615
> URL: https://issues.apache.org/jira/browse/SPARK-24615
> Project: Spark
>  Issue Type: Epic
>  Components: Spark Core
>Affects Versions: 2.4.0
>Reporter: Saisai Shao
>Assignee: Thomas Graves
>Priority: Major
>  Labels: Hydrogen, SPIP
> Attachments: Accelerator-aware scheduling in Apache Spark 3.0.pdf, 
> SPIP_ Accelerator-aware scheduling.pdf
>
>
> (The JIRA received a major update on 2019/02/28. Some comments were based on 
> an earlier version. Please ignore them. New comments start at 
> [#comment-16778026].)
> h2. Background and Motivation
> GPUs and other accelerators have been widely used for accelerating special 
> workloads, e.g., deep learning and signal processing. While users from the AI 
> community use GPUs heavily, they often need Apache Spark to load and process 
> large datasets and to handle complex data scenarios like streaming. YARN and 
> Kubernetes already support GPUs in their recent releases. Although Spark 
> supports those two cluster managers, Spark itself is not aware of GPUs 
> exposed by them and hence Spark cannot properly request GPUs and schedule 
> them for users. This leaves a critical gap to unify big data and AI workloads 
> and make life simpler for end users.
> To make Spark be aware of GPUs, we shall make two major changes at high level:
> * At cluster manager level, we update or upgrade cluster managers to include 
> GPU support. Then we expose user interfaces for Spark to request GPUs from 
> them.
> * Within Spark, we update its scheduler to understand available GPUs 
> allocated to executors, user task requests, and assign GPUs to tasks properly.
> Based on the work done in YARN and Kubernetes to support GPUs and some 
> offline prototypes, we could have necessary features implemented in the next 
> major release of Spark. You can find a detailed scoping doc here, where we 
> listed user stories and their priorities.
> h2. Goals
> * Make Spark 3.0 GPU-aware in standalone, YARN, and Kubernetes.
> * No regression on scheduler performance for normal jobs.
> h2. Non-goals
> * Fine-grained scheduling within one GPU card.
> ** We treat one GPU card and its memory together as a non-divisible unit.
> * Support TPU.
> * Support Mesos.
> * Support Windows.
> h2. Target Personas
> * Admins who need to configure clusters to run Spark with GPU nodes.
> * Data scientists who need to build DL applications on Spark.
> * Developers who need to integrate DL features on Spark.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-30647) When creating a custom datasource File NotFoundExpection happens

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-30647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029788#comment-17029788
 ] 

Jorge Machado commented on SPARK-30647:
---

2.4x has the same issue.

> When creating a custom datasource File NotFoundExpection happens
> 
>
> Key: SPARK-30647
> URL: https://issues.apache.org/jira/browse/SPARK-30647
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Affects Versions: 2.3.2
>    Reporter: Jorge Machado
>Priority: Major
>
> Hello, I'm creating a datasource based on FileFormat and DataSourceRegister. 
> when I pass a path or a file that has a white space it seems to fail wit the 
> error: 
> {code:java}
>  org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in 
> stage 2.0 failed 1 times, most recent failure: Lost task 1.0 in stage 2.0 
> (TID 213, localhost, executor driver): java.io.FileNotFoundException: File 
> file:somePath/0019_leftImg8%20bit.png does not exist It is possible the 
> underlying files have been updated. You can explicitly invalidate the cache 
> in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating 
> the Dataset/DataFrame involved. at 
> org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:127)
>  at 
> org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:177)
>  at 
> org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)
>  at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
>  Source) at 
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
>  at 
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)
>  at 
> org.apache.spark.sql.execution.columnar.CachedRDDBuilder$$anonfun$1$$anon$1.hasNext(InMemoryRelation.scala:125)
>  at 
> org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
>  at 
> org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
>  at 
> org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1165)
>  at 
> org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1156)
>  at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1091) at 
> org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1156)
> {code}
> I'm happy to fix this if someone tells me where I need to look.  
> I think it is on org.apache.spark.rdd.InputFileBlockHolder : 
> {code:java}
> inputBlock.set(new FileBlock(UTF8String.fromString(filePath), startOffset, 
> length))
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-27990) Provide a way to recursively load data from datasource

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-27990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028856#comment-17028856
 ] 

Jorge Machado commented on SPARK-27990:
---

[~nchammas]: Just pass this like: 
{code:java}
//.option("pathGlobFilter", 
".*\\.png|.*\\.jpg|.*\\.jpeg|.*\\.PNG|.*\\.JPG|.*\\.JPEG")
//.option("recursiveFileLookup", "true")
{code}

> Provide a way to recursively load data from datasource
> --
>
> Key: SPARK-27990
> URL: https://issues.apache.org/jira/browse/SPARK-27990
> Project: Spark
>  Issue Type: New Feature
>  Components: ML, SQL
>Affects Versions: 2.4.3
>Reporter: Weichen Xu
>Assignee: Weichen Xu
>Priority: Major
> Fix For: 3.0.0
>
>
> Provide a way to recursively load data from datasource.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-27990) Provide a way to recursively load data from datasource

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-27990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17028854#comment-17028854
 ] 

Jorge Machado commented on SPARK-27990:
---

Can we backport this to 2.4.4 ?

> Provide a way to recursively load data from datasource
> --
>
> Key: SPARK-27990
> URL: https://issues.apache.org/jira/browse/SPARK-27990
> Project: Spark
>  Issue Type: New Feature
>  Components: ML, SQL
>Affects Versions: 2.4.3
>Reporter: Weichen Xu
>Assignee: Weichen Xu
>Priority: Major
> Fix For: 3.0.0
>
>
> Provide a way to recursively load data from datasource.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-30647) When creating a custom datasource File NotFoundExpection happens

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-30647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17024242#comment-17024242
 ] 

Jorge Machado commented on SPARK-30647:
---

I found a way to overcome this. I just replace %20 with " " like this:
{code:java}
(file: PartitionedFile) => {
val origin = file.filePath.
replace("%20", " ")
}{code}

> When creating a custom datasource File NotFoundExpection happens
> 
>
> Key: SPARK-30647
> URL: https://issues.apache.org/jira/browse/SPARK-30647
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Affects Versions: 2.3.2
>Reporter: Jorge Machado
>Priority: Major
>
> Hello, I'm creating a datasource based on FileFormat and DataSourceRegister. 
> when I pass a path or a file that has a white space it seems to fail wit the 
> error: 
> {code:java}
>  org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in 
> stage 2.0 failed 1 times, most recent failure: Lost task 1.0 in stage 2.0 
> (TID 213, localhost, executor driver): java.io.FileNotFoundException: File 
> file:somePath/0019_leftImg8%20bit.png does not exist It is possible the 
> underlying files have been updated. You can explicitly invalidate the cache 
> in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating 
> the Dataset/DataFrame involved. at 
> org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:127)
>  at 
> org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:177)
>  at 
> org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)
>  at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
>  Source) at 
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
>  at 
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)
>  at 
> org.apache.spark.sql.execution.columnar.CachedRDDBuilder$$anonfun$1$$anon$1.hasNext(InMemoryRelation.scala:125)
>  at 
> org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
>  at 
> org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
>  at 
> org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1165)
>  at 
> org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1156)
>  at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1091) at 
> org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1156)
> {code}
> I'm happy to fix this if someone tells me where I need to look.  
> I think it is on org.apache.spark.rdd.InputFileBlockHolder : 
> {code:java}
> inputBlock.set(new FileBlock(UTF8String.fromString(filePath), startOffset, 
> length))
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-23148) spark.read.csv with multiline=true gives FileNotFoundException if path contains spaces

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-23148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023935#comment-17023935
 ] 

Jorge Machado edited comment on SPARK-23148 at 1/27/20 7:22 AM:


Hi [~hyukjin.kwon] and [~henryr]  , I have the same problem if I create a 
custom data source 

```

class ImageFileValidator extends FileFormat with DataSourceRegister with 
Serializable

```

So the Problem Needs to be in some other places. 

Here my trace: 

 

I created https://issues.apache.org/jira/browse/SPARK-30647
{code:java}
 org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in 
stage 2.0 failed 1 times, most recent failure: Lost task 1.0 in stage 2.0 (TID 
213, localhost, executor driver): java.io.FileNotFoundException: File 
file:somePath/0019_leftImg8%20bit.png does not exist
It is possible the underlying files have been updated. You can explicitly 
invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in 
SQL or by recreating the Dataset/DataFrame involved.
   at 
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:127)
   at 
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:177)
   at 
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)
   at 
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
 Source)
   at 
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
   at 
org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)
   at 
org.apache.spark.sql.execution.columnar.CachedRDDBuilder$$anonfun$1$$anon$1.hasNext(InMemoryRelation.scala:125)
   at 
org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
   at 
org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
   at 
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1165)
   at 
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1156)
   at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1091)
   at 
org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1156)

{code}


was (Author: jomach):
Hi [~hyukjin.kwon] and [~henryr]  , I have the same problem if I create a 
custom data source 

```

class ImageFileValidator extends FileFormat with DataSourceRegister with 
Serializable

```

So the Problem Needs to be in some other places. 

Here my trace: 
{code:java}
 org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in 
stage 2.0 failed 1 times, most recent failure: Lost task 1.0 in stage 2.0 (TID 
213, localhost, executor driver): java.io.FileNotFoundException: File 
file:somePath/0019_leftImg8%20bit.png does not exist
It is possible the underlying files have been updated. You can explicitly 
invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in 
SQL or by recreating the Dataset/DataFrame involved.
   at 
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:127)
   at 
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:177)
   at 
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)
   at 
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
 Source)
   at 
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
   at 
org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)
   at 
org.apache.spark.sql.execution.columnar.CachedRDDBuilder$$anonfun$1$$anon$1.hasNext(InMemoryRelation.scala:125)
   at 
org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
   at 
org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
   at 
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1165)
   at 
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1156)
   at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1091)
   at 
org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1156)

{code}

> spark.read.csv with multiline=true gives FileNotFoundException if path 
> contains spaces
> --
>
> Key: SPARK-23148
> URL: https://issues.apache.org/jira/browse/SPARK-23148
>

[jira] [Updated] (SPARK-30647) When creating a custom datasource File NotFoundExpection happens

[Jorge Machado (Jira)]

 [ 
https://issues.apache.org/jira/browse/SPARK-30647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jorge Machado updated SPARK-30647:
--
Issue Type: Bug  (was: Improvement)

> When creating a custom datasource File NotFoundExpection happens
> 
>
> Key: SPARK-30647
> URL: https://issues.apache.org/jira/browse/SPARK-30647
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Affects Versions: 2.3.2
>    Reporter: Jorge Machado
>Priority: Major
>
> Hello, I'm creating a datasource based on FileFormat and DataSourceRegister. 
> when I pass a path or a file that has a white space it seems to fail wit the 
> error: 
> {code:java}
>  org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in 
> stage 2.0 failed 1 times, most recent failure: Lost task 1.0 in stage 2.0 
> (TID 213, localhost, executor driver): java.io.FileNotFoundException: File 
> file:somePath/0019_leftImg8%20bit.png does not exist It is possible the 
> underlying files have been updated. You can explicitly invalidate the cache 
> in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating 
> the Dataset/DataFrame involved. at 
> org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:127)
>  at 
> org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:177)
>  at 
> org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)
>  at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
>  Source) at 
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
>  at 
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)
>  at 
> org.apache.spark.sql.execution.columnar.CachedRDDBuilder$$anonfun$1$$anon$1.hasNext(InMemoryRelation.scala:125)
>  at 
> org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
>  at 
> org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
>  at 
> org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1165)
>  at 
> org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1156)
>  at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1091) at 
> org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1156)
> {code}
> I'm happy to fix this if someone tells me where I need to look.  
> I think it is on org.apache.spark.rdd.InputFileBlockHolder : 
> {code:java}
> inputBlock.set(new FileBlock(UTF8String.fromString(filePath), startOffset, 
> length))
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-30647) When creating a custom datasource File NotFoundExpection happens

[Jorge Machado (Jira)]

Jorge Machado created SPARK-30647:
-

 Summary: When creating a custom datasource File NotFoundExpection 
happens
 Key: SPARK-30647
 URL: https://issues.apache.org/jira/browse/SPARK-30647
 Project: Spark
  Issue Type: Improvement
  Components: Spark Core
Affects Versions: 2.3.2
Reporter: Jorge Machado


Hello, I'm creating a datasource based on FileFormat and DataSourceRegister. 

when I pass a path or a file that has a white space it seems to fail wit the 
error: 
{code:java}
 org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in 
stage 2.0 failed 1 times, most recent failure: Lost task 1.0 in stage 2.0 (TID 
213, localhost, executor driver): java.io.FileNotFoundException: File 
file:somePath/0019_leftImg8%20bit.png does not exist It is possible the 
underlying files have been updated. You can explicitly invalidate the cache in 
Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the 
Dataset/DataFrame involved. at 
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:127)
 at 
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:177)
 at 
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)
 at 
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
 Source) at 
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
 at 
org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)
 at 
org.apache.spark.sql.execution.columnar.CachedRDDBuilder$$anonfun$1$$anon$1.hasNext(InMemoryRelation.scala:125)
 at 
org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221) 
at 
org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
 at 
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1165)
 at 
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1156)
 at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1091) at 
org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1156)
{code}
I'm happy to fix this if someone tells me where I need to look.  

I think it is on org.apache.spark.rdd.InputFileBlockHolder : 
{code:java}
inputBlock.set(new FileBlock(UTF8String.fromString(filePath), startOffset, 
length))
{code}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-23148) spark.read.csv with multiline=true gives FileNotFoundException if path contains spaces

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-23148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023935#comment-17023935
 ] 

Jorge Machado edited comment on SPARK-23148 at 1/26/20 9:29 PM:


Hi [~hyukjin.kwon] , I have the same problem if I create a custom data source 

```

class ImageFileValidator extends FileFormat with DataSourceRegister with 
Serializable

```

So the Problem Needs to be in some other places. 

Here my trace: 
{code:java}
 org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in 
stage 2.0 failed 1 times, most recent failure: Lost task 1.0 in stage 2.0 (TID 
213, localhost, executor driver): java.io.FileNotFoundException: File 
file:somePath/0019_leftImg8%20bit.png does not exist
It is possible the underlying files have been updated. You can explicitly 
invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in 
SQL or by recreating the Dataset/DataFrame involved.
   at 
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:127)
   at 
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:177)
   at 
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)
   at 
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
 Source)
   at 
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
   at 
org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)
   at 
org.apache.spark.sql.execution.columnar.CachedRDDBuilder$$anonfun$1$$anon$1.hasNext(InMemoryRelation.scala:125)
   at 
org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
   at 
org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
   at 
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1165)
   at 
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1156)
   at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1091)
   at 
org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1156)

{code}


was (Author: jomach):
So I have the same problem if I create a custom data source 

```

class ImageFileValidator extends FileFormat with DataSourceRegister with 
Serializable

```

So the Problem Needs to be in some other places. 

Here my trace: 
{code:java}
 org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in 
stage 2.0 failed 1 times, most recent failure: Lost task 1.0 in stage 2.0 (TID 
213, localhost, executor driver): java.io.FileNotFoundException: File 
file:somePath/0019_leftImg8%20bit.png does not exist
It is possible the underlying files have been updated. You can explicitly 
invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in 
SQL or by recreating the Dataset/DataFrame involved.
   at 
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:127)
   at 
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:177)
   at 
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)
   at 
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
 Source)
   at 
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
   at 
org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)
   at 
org.apache.spark.sql.execution.columnar.CachedRDDBuilder$$anonfun$1$$anon$1.hasNext(InMemoryRelation.scala:125)
   at 
org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
   at 
org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
   at 
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1165)
   at 
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1156)
   at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1091)
   at 
org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1156)

{code}

> spark.read.csv with multiline=true gives FileNotFoundException if path 
> contains spaces
> --
>
> Key: SPARK-23148
> URL: https://issues.apache.org/jira/browse/SPARK-23148
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Version

[jira] [Comment Edited] (SPARK-23148) spark.read.csv with multiline=true gives FileNotFoundException if path contains spaces

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-23148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023935#comment-17023935
 ] 

Jorge Machado edited comment on SPARK-23148 at 1/26/20 9:29 PM:


Hi [~hyukjin.kwon] and [~henryr]  , I have the same problem if I create a 
custom data source 

```

class ImageFileValidator extends FileFormat with DataSourceRegister with 
Serializable

```

So the Problem Needs to be in some other places. 

Here my trace: 
{code:java}
 org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in 
stage 2.0 failed 1 times, most recent failure: Lost task 1.0 in stage 2.0 (TID 
213, localhost, executor driver): java.io.FileNotFoundException: File 
file:somePath/0019_leftImg8%20bit.png does not exist
It is possible the underlying files have been updated. You can explicitly 
invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in 
SQL or by recreating the Dataset/DataFrame involved.
   at 
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:127)
   at 
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:177)
   at 
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)
   at 
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
 Source)
   at 
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
   at 
org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)
   at 
org.apache.spark.sql.execution.columnar.CachedRDDBuilder$$anonfun$1$$anon$1.hasNext(InMemoryRelation.scala:125)
   at 
org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
   at 
org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
   at 
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1165)
   at 
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1156)
   at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1091)
   at 
org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1156)

{code}


was (Author: jomach):
Hi [~hyukjin.kwon] , I have the same problem if I create a custom data source 

```

class ImageFileValidator extends FileFormat with DataSourceRegister with 
Serializable

```

So the Problem Needs to be in some other places. 

Here my trace: 
{code:java}
 org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in 
stage 2.0 failed 1 times, most recent failure: Lost task 1.0 in stage 2.0 (TID 
213, localhost, executor driver): java.io.FileNotFoundException: File 
file:somePath/0019_leftImg8%20bit.png does not exist
It is possible the underlying files have been updated. You can explicitly 
invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in 
SQL or by recreating the Dataset/DataFrame involved.
   at 
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:127)
   at 
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:177)
   at 
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)
   at 
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
 Source)
   at 
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
   at 
org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)
   at 
org.apache.spark.sql.execution.columnar.CachedRDDBuilder$$anonfun$1$$anon$1.hasNext(InMemoryRelation.scala:125)
   at 
org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
   at 
org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
   at 
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1165)
   at 
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1156)
   at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1091)
   at 
org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1156)

{code}

> spark.read.csv with multiline=true gives FileNotFoundException if path 
> contains spaces
> --
>
> Key: SPARK-23148
> URL: https://issues.apache.org/jira/browse/SPARK-23148
> Project: Spark
>  Issue Type: Bug
>  Components

[jira] [Commented] (SPARK-23148) spark.read.csv with multiline=true gives FileNotFoundException if path contains spaces

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-23148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17023935#comment-17023935
 ] 

Jorge Machado commented on SPARK-23148:
---

So I have the same problem if I create a custom data source 

```

class ImageFileValidator extends FileFormat with DataSourceRegister with 
Serializable

```

So the Problem Needs to be in some other places. 

Here my trace: 
{code:java}
 org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in 
stage 2.0 failed 1 times, most recent failure: Lost task 1.0 in stage 2.0 (TID 
213, localhost, executor driver): java.io.FileNotFoundException: File 
file:somePath/0019_leftImg8%20bit.png does not exist
It is possible the underlying files have been updated. You can explicitly 
invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in 
SQL or by recreating the Dataset/DataFrame involved.
   at 
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:127)
   at 
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:177)
   at 
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:101)
   at 
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown
 Source)
   at 
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
   at 
org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636)
   at 
org.apache.spark.sql.execution.columnar.CachedRDDBuilder$$anonfun$1$$anon$1.hasNext(InMemoryRelation.scala:125)
   at 
org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:221)
   at 
org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:299)
   at 
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1165)
   at 
org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1156)
   at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1091)
   at 
org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1156)

{code}

> spark.read.csv with multiline=true gives FileNotFoundException if path 
> contains spaces
> --
>
> Key: SPARK-23148
> URL: https://issues.apache.org/jira/browse/SPARK-23148
> Project: Spark
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 2.3.0
>Reporter: Bogdan Raducanu
>Assignee: Henry Robinson
>Priority: Major
> Fix For: 2.3.0
>
>
> Repro code:
> {code:java}
> spark.range(10).write.csv("/tmp/a b c/a.csv")
> spark.read.option("multiLine", false).csv("/tmp/a b c/a.csv").count
> 10
> spark.read.option("multiLine", true).csv("/tmp/a b c/a.csv").count
> java.io.FileNotFoundException: File 
> file:/tmp/a%20b%20c/a.csv/part-0-cf84f9b2-5fe6-4f54-a130-a1737689db00-c000.csv
>  does not exist
> {code}
> Trying to manually escape fails in a different place:
> {code}
> spark.read.option("multiLine", true).csv("/tmp/a%20b%20c/a.csv").count
> org.apache.spark.sql.AnalysisException: Path does not exist: 
> file:/tmp/a%20b%20c/a.csv;
>   at 
> org.apache.spark.sql.execution.datasources.DataSource$.org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary(DataSource.scala:683)
>   at 
> org.apache.spark.sql.execution.datasources.DataSource$$anonfun$15.apply(DataSource.scala:387)
>   at 
> org.apache.spark.sql.execution.datasources.DataSource$$anonfun$15.apply(DataSource.scala:387)
>   at 
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
>   at 
> scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:241)
>   at scala.collection.immutable.List.foreach(List.scala:381)
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-29158) Expose SerializableConfiguration for DSv2

[Jorge Machado (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-29158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997947#comment-16997947
 ] 

Jorge Machado commented on SPARK-29158:
---

How can we get SerializableConfiguration with 2.4.4 ? Any alternative ?

> Expose SerializableConfiguration for DSv2
> -
>
> Key: SPARK-29158
> URL: https://issues.apache.org/jira/browse/SPARK-29158
> Project: Spark
>  Issue Type: Improvement
>  Components: Spark Core, SQL
>Affects Versions: 2.4.5, 3.0.0
>Reporter: Holden Karau
>Assignee: Holden Karau
>Priority: Major
> Fix For: 3.0.0
>
>
> Since we use it frequently inside of our own DataSourceV2 implementations (13 
> times from `
>  grep -r broadcastedConf ./sql/core/src/ |grep val |wc -l`
> ) we should expose the SerializableConfiguration for DSv2 dev work



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (AIRFLOW-6132) AzureContainerInstancesOperator should allow to pass in tags

[Jorge Machado (Jira)]

Jorge Machado created AIRFLOW-6132:
--

 Summary: AzureContainerInstancesOperator should allow to pass in 
tags
 Key: AIRFLOW-6132
 URL: https://issues.apache.org/jira/browse/AIRFLOW-6132
 Project: Apache Airflow
  Issue Type: Improvement
  Components: operators
Affects Versions: 1.10.4
Reporter: Jorge Machado


Hi, 

 

Currently I'm using airflow with AzureContainerInstancesOperator, I need to add 
some tags on the containers that I start. This is currently not possible. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HDFS-9924) [umbrella] Nonblocking HDFS Access

[Jorge Machado (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-9924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900426#comment-16900426
 ] 

Jorge Machado commented on HDFS-9924:
-

Any ideas how to improve the hdfs dfs put command with async io ? Or I'm I in 
the wrong thread. From what I saw here 
[https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java#L129]
 Copy Commands still uses blocking io right ?

> [umbrella] Nonblocking HDFS Access
> --
>
> Key: HDFS-9924
> URL: https://issues.apache.org/jira/browse/HDFS-9924
> Project: Hadoop HDFS
>  Issue Type: New Feature
>  Components: fs
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Duo Zhang
>Priority: Major
> Attachments: Async-HDFS-Performance-Report.pdf, 
> AsyncHdfs20160510.pdf, HDFS-9924-POC.patch
>
>
> This is an umbrella JIRA for supporting Nonblocking HDFS Access.
> Currently, all the API methods are blocking calls -- the caller is blocked 
> until the method returns.  It is very slow if a client makes a large number 
> of independent calls in a single thread since each call has to wait until the 
> previous call is finished.  It is inefficient if a client needs to create a 
> large number of threads to invoke the calls.
> We propose adding a new API to support nonblocking calls, i.e. the caller is 
> not blocked.  The methods in the new API immediately return a Java Future 
> object.  The return value can be obtained by the usual Future.get() method.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

[jira] [Commented] (HDFS-916) Rewrite DFSOutputStream to use a single thread with NIO

[Jorge Machado (JIRA)]

[ 
https://issues.apache.org/jira/browse/HDFS-916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900421#comment-16900421
 ] 

Jorge Machado commented on HDFS-916:


Hi Guys, I know this is pretty old but is there any status on this ? We are 
transferring like 30TB via hdfs dfs copyFromLocal to a hadoop Cluster, 
Currently we have the cpus as bottleneck...  

> Rewrite DFSOutputStream to use a single thread with NIO
> ---
>
> Key: HDFS-916
> URL: https://issues.apache.org/jira/browse/HDFS-916
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: hdfs-client
>Affects Versions: 0.22.0
>Reporter: Todd Lipcon
>Priority: Major
>
> The DFS write pipeline code has some really hairy multi-threaded 
> synchronization. There have been a lot of bugs produced by this (HDFS-101, 
> HDFS-793, HDFS-915, tens of others) since it's very hard to understand the 
> message passing, lock sharing, and interruption properties. The reason for 
> the multiple threads is to be able to simultaneously send and receive. If 
> instead of using multiple threads, it used nonblocking IO, I think the whole 
> thing would be a lot less error prone.
> I think we could do this in two halves: one half is the DFSOutputStream. The 
> other half is BlockReceiver. I opened this JIRA first as I think it's simpler 
> (only one TCP connection to deal with, rather than an up and downstream)
> Opinions? Am I crazy? I would like to see some agreement on the idea before I 
> spend time writing code.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

Restarting mesas-agent kills executors

[Jorge Machado]

Hi Guys, 

I was reading about agent restarts on 
http://mesos.apache.org/documentation/latest/agent-recovery/ 
<http://mesos.apache.org/documentation/latest/agent-recovery/>
From what I understood, If I had a task running and we restart the mesos-agent 
I should not loose any task running. 
This is not the case for systemctl (or with service command) from ubuntu 18.04. 
Our Framework has checkpointing active...

My config: 

[Unit]
Description=Mesos Agent
After=network.target
Wants=network.target

[Service]
Environment=LIBPROCESS_SSL_ENABLED=true
Environment=LIBPROCESS_SSL_SUPPORT_DOWNGRADE=false
Environment=LIBPROCESS_SSL_CIPHERS=AES128-SHA:AES256-SHA:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA
Environment=LIBPROCESS_SSL_KEY_FILE=/etc/ssl/private/server_2048.key
Environment=LIBPROCESS_SSL_CERT_FILE=/etc/ssl/server.crt
Environment=LIBPROCESS_SSL_CA_FILE=/etc/pki/trust/anchors/it4ad.pem

ExecStart=/usr/local/sbin/mesos-agent \
--master= \
--work_dir=/data/mesos/work \
--log_dir=/var/log/mesos \
--executor_registration_timeout=20mins \
--executor_environment_variables=file:///etc/mesos/executor_envs.json \
--resources=file:///etc/mesos/resources.txt \
--image_gc_config=file:///etc/mesos/image-gc-config.json \

--isolation=cgroups/cpu,cgroups/mem,cgroups/devices,filesystem/linux,gpu/nvidia,docker/runtime,namespaces/pid,namespaces/ipc
 \
--image_providers=docker \
--docker_store_dir=/data/mesos/store/docker \
--gc_delay=3weeks \
--attributes=

KillMode=control-cgroup
Restart=always
RestartSec=20
LimitNOFILE=infinity
CPUAccounting=true
MemoryAccounting=true
TasksMax=infinity

[Install]
WantedBy=multi-user.target



Any tipp ? thx



Jorge Machado
www.jmachado.me

[jira] [Commented] (SPARK-24615) SPIP: Accelerator-aware task scheduling for Spark

[Jorge Machado (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882686#comment-16882686
 ] 

Jorge Machado commented on SPARK-24615:
---

Hi Guys, is there any progress here ? I would like to help on the 
implementation step. I did not see any interfaces designed yet.

> SPIP: Accelerator-aware task scheduling for Spark
> -
>
> Key: SPARK-24615
> URL: https://issues.apache.org/jira/browse/SPARK-24615
> Project: Spark
>  Issue Type: Epic
>  Components: Spark Core
>Affects Versions: 2.4.0
>Reporter: Saisai Shao
>Assignee: Thomas Graves
>Priority: Major
>  Labels: Hydrogen, SPIP
> Attachments: Accelerator-aware scheduling in Apache Spark 3.0.pdf, 
> SPIP_ Accelerator-aware scheduling.pdf
>
>
> (The JIRA received a major update on 2019/02/28. Some comments were based on 
> an earlier version. Please ignore them. New comments start at 
> [#comment-16778026].)
> h2. Background and Motivation
> GPUs and other accelerators have been widely used for accelerating special 
> workloads, e.g., deep learning and signal processing. While users from the AI 
> community use GPUs heavily, they often need Apache Spark to load and process 
> large datasets and to handle complex data scenarios like streaming. YARN and 
> Kubernetes already support GPUs in their recent releases. Although Spark 
> supports those two cluster managers, Spark itself is not aware of GPUs 
> exposed by them and hence Spark cannot properly request GPUs and schedule 
> them for users. This leaves a critical gap to unify big data and AI workloads 
> and make life simpler for end users.
> To make Spark be aware of GPUs, we shall make two major changes at high level:
> * At cluster manager level, we update or upgrade cluster managers to include 
> GPU support. Then we expose user interfaces for Spark to request GPUs from 
> them.
> * Within Spark, we update its scheduler to understand available GPUs 
> allocated to executors, user task requests, and assign GPUs to tasks properly.
> Based on the work done in YARN and Kubernetes to support GPUs and some 
> offline prototypes, we could have necessary features implemented in the next 
> major release of Spark. You can find a detailed scoping doc here, where we 
> listed user stories and their priorities.
> h2. Goals
> * Make Spark 3.0 GPU-aware in standalone, YARN, and Kubernetes.
> * No regression on scheduler performance for normal jobs.
> h2. Non-goals
> * Fine-grained scheduling within one GPU card.
> ** We treat one GPU card and its memory together as a non-divisible unit.
> * Support TPU.
> * Support Mesos.
> * Support Windows.
> h2. Target Personas
> * Admins who need to configure clusters to run Spark with GPU nodes.
> * Data scientists who need to build DL applications on Spark.
> * Developers who need to integrate DL features on Spark.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

How to run spark on GPUs

[Jorge Machado]

Hi Guys, 

what is the current recommend way to use GPUs on spark ? 

Which scheduler should we use ? Mesos Or Kubernetes ? 

What are the approaches to follow until 
https://issues.apache.org/jira/browse/SPARK-24615 
 is in place. 

Thanks
Jorge 

Re: [VOTE] Release Apache Mesos 1.8.0 (rc3)

[Jorge Machado]

Hello Benno, 

+1 

Tested on : 
Ubuntu 18.04 with SSL 
8 GPUs per server
 NVIDIA-SMI 418.56
Tested gpu workload with: tensorflow
Image used for testing: tensorflow/tensorflow:1.13.1-gpu-py3
Result: 
--- versions --
DISTRIB_ID=Ubuntu
VERSION_ID="16.04"
driver_version
418.56
CUDA Version 10.0.130
tf version: 1.13.1
---
TensorFlow:  1.13
Model:   resnet50
Dataset: imagenet (synthetic)
Mode:training
SingleSess:  False
Batch size:  32 global
 32 per device
Num batches: 500
Num epochs:  0.01
Devices: ['/gpu:0']
NUMA bind:   False
Data format: NCHW
Optimizer:   sgd
Variables:   parameter_server
==
Generating training model
Initializing graph
Running warm up
Done warm up
...
Executing pre-exec command 
'{"arguments":["ln","-s","/sys/fs/cgroup/cpu,cpuacct","/data0/mesos/work/provisioner/containers/3b1ccd4e-e2d6-44ba-bf8d-f7b29881f6a6/backends/overlay/rootfses/e06cb46b-07e6-4e87-8b2d-fa9af29e298b/sys/fs/cgroup/cpuacct"],"shell":false,"value":"ln"}'
Executing pre-exec command 
'{"arguments":["ln","-s","/sys/fs/cgroup/cpu,cpuacct","/data0/mesos/work/provisioner/containers/3b1ccd4e-e2d6-44ba-bf8d-f7b29881f6a6/backends/overlay/rootfses/e06cb46b-07e6-4e87-8b2d-fa9af29e298b/sys/fs/cgroup/cpu"],"shell":false,"value":"ln"}'
Changing root to 
/data0/mesos/work/provisioner/containers/3b1ccd4e-e2d6-44ba-bf8d-f7b29881f6a6/backends/overlay/rootfses/e06cb46b-07e6-4e87-8b2d-fa9af29e298b
2019-05-02 07:16:57.039394: I 
tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports 
instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-05-02 07:16:57.250080: I tensorflow/compiler/xla/service/service.cc:150] 
XLA service 0x4ec62d0 executing computations on platform CUDA. Devices:
2019-05-02 07:16:57.250152: I tensorflow/compiler/xla/service/service.cc:158]   
StreamExecutor device (0): Tesla V100-PCIE-16GB, Compute Capability 7.0
2019-05-02 07:16:57.273117: I 
tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 
259420 Hz
2019-05-02 07:16:57.277123: I tensorflow/compiler/xla/service/service.cc:150] 
XLA service 0x503da70 executing computations on platform Host. Devices:
2019-05-02 07:16:57.277177: I tensorflow/compiler/xla/service/service.cc:158]   
StreamExecutor device (0): , 
2019-05-02 07:16:57.278024: I 
tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with 
properties: 
name: Tesla V100-PCIE-16GB major: 7 minor: 0 memoryClockRate(GHz): 1.38
pciBusID: :83:00.0
totalMemory: 15.75GiB freeMemory: 15.44GiB
2019-05-02 07:16:57.278046: I 
tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu 
devices: 0

> On 29 Apr 2019, at 22:05, Benno Evers  wrote:
> 
> Hi Jorge,
> 
> I'm admittedly not too familiar with CUDA and tensorflow but the error 
> message you describe sounds to me more like a build issue, i.e. it sounds 
> like the version of the nvidia driver is different between the docker image 
> and the host system?
> 
> Maybe you could continue investigating to see if this is related to the 
> release itself or caused by some external cause, and create a JIRA ticket to 
> capture your findings?
> 
> Thanks,
> Benno
> 
> On Fri, Apr 26, 2019 at 9:55 PM Jorge Machado  <mailto:jom...@me.com>> wrote:
> Hi all, 
> 
> did someone tested it on ubuntu 18.04 + nvidia-docker2 ? We are having some 
> issues using the cuda 10+ images when doing real processing. We still need to 
> check some things but basically we get: 
> kernel version 418.56.0 does not match DSO version 410.48.0 -- cannot find 
> working devices in this configuration
> 
> Logs:
> I0424 13:27:14.00058630 executor.cpp:726] Forked command at 73
> Preparing rootfs at 
> '/data0/mesos/work/provisioner/containers/548d3cae-30b5-4530-a8db-f94b00215718/backends/overlay/rootfses/e1ceb89e-3abc-4587-a87c-d63037b7ae8b'
> Marked '/' as rslave
> Executing pre-exec command 
> '{"arguments":["ln","-s","/sys/fs/cgroup/cpu,cpuacct","/data0/mesos/work/provisioner/containers/548d3cae-30b5-4530-a8db-f94b00215718/backends/overlay/rootfses/e1ceb89e-3abc-4587-a87c-d63037b7ae8b/sys/fs/cgroup/cpuacct"],"shell":false,"value":"ln"}'
> Executing pre-exec command 
> '{"arguments":["ln","-s","/sys/fs/cgroup/cpu,cpuacct","/data0/mesos/work/provisioner/containers/548d3cae-30b5-4530-a8db-f94b00215718/backends/overlay/rootfses/e1ceb89e-3abc-4587-a87c-d63037b7ae8b/sys/fs/cgroup/cpu"],"shell":false,"value":"ln"}'
> Changing root to 
> /data0/mesos/work/provisioner/containers/548d3cae-30b5-4530-a8db-f94b00215

Re: [VOTE] Release Apache Mesos 1.8.0 (rc3)

[Jorge Machado]

Hi all, 

did someone tested it on ubuntu 18.04 + nvidia-docker2 ? We are having some 
issues using the cuda 10+ images when doing real processing. We still need to 
check some things but basically we get: 
kernel version 418.56.0 does not match DSO version 410.48.0 -- cannot find 
working devices in this configuration

Logs:
I0424 13:27:14.00058630 executor.cpp:726] Forked command at 73
Preparing rootfs at 
'/data0/mesos/work/provisioner/containers/548d3cae-30b5-4530-a8db-f94b00215718/backends/overlay/rootfses/e1ceb89e-3abc-4587-a87c-d63037b7ae8b'
Marked '/' as rslave
Executing pre-exec command 
'{"arguments":["ln","-s","/sys/fs/cgroup/cpu,cpuacct","/data0/mesos/work/provisioner/containers/548d3cae-30b5-4530-a8db-f94b00215718/backends/overlay/rootfses/e1ceb89e-3abc-4587-a87c-d63037b7ae8b/sys/fs/cgroup/cpuacct"],"shell":false,"value":"ln"}'
Executing pre-exec command 
'{"arguments":["ln","-s","/sys/fs/cgroup/cpu,cpuacct","/data0/mesos/work/provisioner/containers/548d3cae-30b5-4530-a8db-f94b00215718/backends/overlay/rootfses/e1ceb89e-3abc-4587-a87c-d63037b7ae8b/sys/fs/cgroup/cpu"],"shell":false,"value":"ln"}'
Changing root to 
/data0/mesos/work/provisioner/containers/548d3cae-30b5-4530-a8db-f94b00215718/backends/overlay/rootfses/e1ceb89e-3abc-4587-a87c-d63037b7ae8b
2019-04-24 13:27:18.346994: I 
tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports 
instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-04-24 13:27:18.352203: E 
tensorflow/stream_executor/cuda/cuda_driver.cc:300] failed call to cuInit: 
CUDA_ERROR_UNKNOWN: unknown error
2019-04-24 13:27:18.352243: I 
tensorflow/stream_executor/cuda/cuda_diagnostics.cc:161] retrieving CUDA 
diagnostic information for host: __host__
2019-04-24 13:27:18.352252: I 
tensorflow/stream_executor/cuda/cuda_diagnostics.cc:168] hostname: __host__
2019-04-24 13:27:18.352295: I 
tensorflow/stream_executor/cuda/cuda_diagnostics.cc:192] libcuda reported 
version is: 410.48.0
2019-04-24 13:27:18.352329: I 
tensorflow/stream_executor/cuda/cuda_diagnostics.cc:196] kernel reported 
version is: 418.56.0
2019-04-24 13:27:18.352338: E 
tensorflow/stream_executor/cuda/cuda_diagnostics.cc:306] kernel version 
418.56.0 does not match DSO version 410.48.0 -- cannot find working devices in 
this configuration
2019-04-24 13:27:18.374940: I 
tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 
259392 Hz
2019-04-24 13:27:18.378793: I tensorflow/compiler/xla/service/service.cc:150] 
XLA service 0x4f41e10 executing computations on platform Host. Devices:
2019-04-24 13:27:18.378821: I tensorflow/compiler/xla/service/service.cc:158]   
StreamExecutor device (0): , 
W0424 13:27:18.385210 140191267731200 deprecation.py:323] From 
/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py:263:
 colocate_with (from tensorflow.python.framework.ops) is deprecated and will be 
removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
W0424 13:27:18.399287 140191267731200 deprecation.py:323] From 
/user/tf-benchmarks-113/scripts/tf_cnn_benchmarks/convnet_builder.py:129: 
conv2d (from tensorflow.python.layers.convolutional) is deprecated and will be 
removed in a future version.
Instructions for updating:
Use keras.layers.conv2d instead.
W0424 13:27:18.433226 140191267731200 deprecation.py:323] From 
/user/tf-benchmarks-113/scripts/tf_cnn_benchmarks/convnet_builder.py:261: 
max_pooling2d (from tensorflow.python.layers.pooling) is deprecated and will be 
removed in a future version.
Instructions for updating:
Use keras.layers.max_pooling2d instead.
W0424 13:27:20.197937 140191267731200 deprecation.py:323] From 
/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/losses/losses_impl.py:209:
 to_float (from tensorflow.python.ops.math_ops) is deprecated and will be 
removed in a future version.
Instructions for updating:
Use tf.cast instead.
W0424 13:27:20.312573 140191267731200 deprecation.py:323] From 
/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py:3066: 
to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be 
removed in a future version.
Instructions for updating:
Use tf.cast instead.
W0424 13:27:21.082763 140191267731200 deprecation.py:323] From 
/user/tf-benchmarks-113/scripts/tf_cnn_benchmarks/benchmark_cnn.py:2238: 
__init__ (from tensorflow.python.training.supervisor) is deprecated and will be 
removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
I0424 13:27:22.013817 140191267731200 session_manager.py:491] Running 
local_init_op.
I0424 13:27:22.193911 140191267731200 session_manager.py:493] Done running 
local_init_op.
2019-04-24 13:27:23.181740: E tensorflow/core/common_runtime/executor.cc:624] 
Executor failed to create kernel. Invalid argument: Default MaxPoolingOp only 
supports NHWC on device type CPU
 [[{{node tower_0/v/cg/mpool0/MaxPool}}]]

[jira] [Commented] (MESOS-9740) Invalid protobuf unions in ExecutorInfo::ContainerInfo will prevent agents from reregistering with 1.8+ masters

[Jorge Machado (JIRA)]

[ 
https://issues.apache.org/jira/browse/MESOS-9740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824829#comment-16824829
 ] 

Jorge Machado commented on MESOS-9740:
--

we are running mesos 1.7.1 and have one slave with ubuntu 18.04 wich has the 
master version compiled. We are only using the mesos containerizer and not the 
docker. It works fine for us.

> Invalid protobuf unions in ExecutorInfo::ContainerInfo will prevent agents 
> from reregistering with 1.8+ masters
> ---
>
> Key: MESOS-9740
> URL: https://issues.apache.org/jira/browse/MESOS-9740
> Project: Mesos
>  Issue Type: Bug
>Affects Versions: 1.8.0
>Reporter: Joseph Wu
>Assignee: Benno Evers
>Priority: Blocker
>  Labels: foundations, mesosphere
>
> As part of MESOS-6874, the master now validates protobuf unions passed as 
> part of an {{ExecutorInfo::ContainerInfo}}.  This prevents a task from 
> specifying, for example, a {{ContainerInfo::MESOS}}, but filling out the 
> {{docker}} field (which is then ignored by the agent).
> However, if a task was already launched with an invalid protobuf union, the 
> same validation will happen when the agent tries to reregister with the 
> master.  In this case, if the master is upgraded to validate protobuf unions, 
> the agent reregistration will be rejected.
> {code}
> master.cpp:7201] Dropping re-registration of agent at 
> slave(1)@172.31.47.126:5051 because it sent an invalid re-registration: 
> Protobuf union `mesos.ContainerInfo` with `Type == MESOS` should not have the 
> field `docker` set.
> {code}
> This bug was found when upgrading a 1.7.x test cluster to 1.8.0.  When 
> MESOS-6874 was committed, I had assumed the invalid protobufs would be rare.  
> However, on the test cluster, 13/17 agents had at least one invalid 
> ContainerInfo when reregistering.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Mesos on ssl

[Jorge Machado]

Thanks Hsiao, I think it is this, as a build from master works fine for me. 
When are we releasing 1.8.0 ? 



> On 5 Apr 2019, at 16:52, Chun-Hung Hsiao  wrote:
> 
> I'm not sure if this is related:
> https://issues.apache.org/jira/browse/MESOS-7076
> 
> In summary, Ubuntu 18.04 ships libevent 2.1.x (for OpenSSL 1.1.x support).
> But libevent 2.1.x has an unknown bug that caused some Mesos tests to fail.
> As a workaround, the current Mesos master branch (will be 1.8 soon) bundled
> libevent 2.0.x with a magic patch from Debian 8 for OpenSSL 1.1.x). So
> Mesos 1.8 will be the first official release supporting SSL on Ubuntu 18.04.
> 
> That said, I'm not sure what you encountered is exactly the same bug that
> caused the Mesos tests to fail though. Just a guess ;)
> 
> On Fri, Apr 5, 2019, 12:58 AM Jorge Machado  wrote:
> 
>> Hi Guys,
>> 
>> I'm having issues with mesos versions from tar.gz compared with a build
>> from git master when using ssl.
>> With a build from git ssl agent is fine and for example the endpoint
>> https://mesos-agent:5051/ returns a 404 which is fine.
>> With a build from tar.gz (1.7.1 or 1.7.2) the same endpoint does not work
>> and it just hangs. No logs nothing...
>> I'm testing this on ubuntu 18.04.
>> 
>> Any tipps ?
>> thanks
>> Jorge
>> 
>> 
>> Jorge Machado
>> www.jmachado.me
>> 
>> 
>> 
>> 
>> 
>> 

Mesos on ssl

[Jorge Machado]

Hi Guys, 

I'm having issues with mesos versions from tar.gz compared with a build from 
git master when using ssl. 
With a build from git ssl agent is fine and for example the endpoint 
https://mesos-agent:5051/ returns a 404 which is fine. 
With a build from tar.gz (1.7.1 or 1.7.2) the same endpoint does not work and 
it just hangs. No logs nothing... 
I'm testing this on ubuntu 18.04. 

Any tipps ?
thanks
Jorge


Jorge Machado
www.jmachado.me

[jira] [Commented] (MESOS-6851) make install fails the second time

[Jorge Machado (JIRA)]

[ 
https://issues.apache.org/jira/browse/MESOS-6851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808550#comment-16808550
 ] 

Jorge Machado commented on MESOS-6851:
--

ah nice link. I see that the first one is for Centos anyone for ubuntu 18.04 ? 
Mesosphere only has for ubuntu 16.04

> make install fails the second time
> --
>
> Key: MESOS-6851
> URL: https://issues.apache.org/jira/browse/MESOS-6851
> Project: Mesos
>  Issue Type: Bug
>  Components: build
>Reporter: James Peach
>Priority: Major
>
> Run {{make install}} twice and the second time will fail when it tries to 
> overwrite symlinks:
> {code}
> make[4]: Entering directory '/home/jpeach/upstream/mesos/build/src'
> cd //opt/mesos/etc/mesos && \
>   ln -s mesos-agent-env.sh.template mesos-slave-env.sh.template
> ln: failed to create symbolic link 'mesos-slave-env.sh.template': File exists
> Makefile:12952: recipe for target 'copy-template-and-create-symlink' failed
> make[4]: *** [copy-template-and-create-symlink] Error 1
> make[4]: Leaving directory '/home/jpeach/upstream/mesos/build/src'
> Makefile:12487: recipe for target 'install-data-am' failed
> make[3]: *** [install-data-am] Error 2
> make[3]: Leaving directory '/home/jpeach/upstream/mesos/build/src'
> Makefile:12197: recipe for target 'install-am' failed
> make[2]: *** [install-am] Error 2
> make[2]: Leaving directory '/home/jpeach/upstream/mesos/build/src'
> Makefile:12191: recipe for target 'install' failed
> make[1]: *** [install] Error 2
> make[1]: Leaving directory '/home/jpeach/upstream/mesos/build/src'
> Makefile:764: recipe for target 'install-recursive' failed
> make: *** [install-recursive] Error 1
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Why does not mesos provide linux packages ?

[Jorge Machado]

Hi Guys, 

why don't we have packages for the main ubuntu distributions ? like ubuntu and 
redhat ? 
I have the feeling that everyone is building and creating this packages to 
distribute. Is there a way that we could improve this ? 

thanks 

[jira] [Comment Edited] (MESOS-6851) make install fails the second time

[Jorge Machado (JIRA)]

[ 
https://issues.apache.org/jira/browse/MESOS-6851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808527#comment-16808527
 ] 

Jorge Machado edited comment on MESOS-6851 at 4/3/19 9:32 AM:
--

So I try it right now and it does not fit. It would much easier to provide deb 
and rpms from the project... 

My commands on Jenkins on a container ubuntu:1804: 
{code:java}
sh("git clone https://github.com/mesosphere/mesos-deb-packaging.git | true ")
dir("mesos-deb-packaging"){
echo("Uninstalling existing installation")
sh("cd mesos-repo/build && make uninstall | true")
echo("The next step is going to take 5 min for cloning the repo")
sh("./build_mesos --repo https://github.com/apache/mesos?tag=${MESOS_VERSION} 
--configure-flags '--enable-ssl --enable-libevent --disable-python 
--disable-java'")
}{code}
Logs: 
{code:java}
make  install-exec-hook

make[4]: Entering directory 
'/var/lib/jenkins/workspace/user/machjor/build_rpm_mesos_machjor@2/mesos-deb-packaging/mesos-repo/build/src'
cd 
/var/lib/jenkins/workspace/user/machjor/build_rpm_mesos_machjor@2/mesos-deb-packaging/mesos-repo/toor//usr/etc/mesos
 && \
  ln -s mesos-agent-env.sh.template mesos-slave-env.sh.template
ln: failed to create symbolic link 'mesos-slave-env.sh.template': File exists
Makefile:14819: recipe for target 'copy-template-and-create-symlink' failed
make[4]: *** [copy-template-and-create-symlink] Error 1
make[4]: Leaving directory 
'/var/lib/jenkins/workspace/user//build_rpm_mesos_@2/mesos-deb-packaging/mesos-repo/build/src'
Makefile:14338: recipe for target 'install-data-am' failed
{code}
Any advice ? 


was (Author: jomach):
So I try it right now and it does not fit. It would much easier to provide deb 
and rpms from the project... 

My commands on Jenkins on a container ubuntu:1804: 
{code:java}
sh("git clone https://github.com/mesosphere/mesos-deb-packaging.git | true ")
dir("mesos-deb-packaging"){
echo("Uninstalling existing installation")
sh("cd mesos-repo/build && make uninstall | true")
echo("The next step is going to take 5 min for cloning the repo")
sh("./build_mesos --repo https://github.com/apache/mesos?tag=${MESOS_VERSION} 
--configure-flags '--enable-ssl --enable-libevent --disable-python 
--disable-java'")
}{code}
Logs: 
{code:java}
make  install-exec-hook

make[4]: Entering directory 
'/var/lib/jenkins/workspace/user/machjor/build_rpm_mesos_machjor@2/mesos-deb-packaging/mesos-repo/build/src'
cd 
/var/lib/jenkins/workspace/user/machjor/build_rpm_mesos_machjor@2/mesos-deb-packaging/mesos-repo/toor//usr/etc/mesos
 && \
  ln -s mesos-agent-env.sh.template mesos-slave-env.sh.template
ln: failed to create symbolic link 'mesos-slave-env.sh.template': File exists
Makefile:14819: recipe for target 'copy-template-and-create-symlink' failed
make[4]: *** [copy-template-and-create-symlink] Error 1
make[4]: Leaving directory 
'/var/lib/jenkins/workspace/user/machjor/build_rpm_mesos_machjor@2/mesos-deb-packaging/mesos-repo/build/src'
Makefile:14338: recipe for target 'install-data-am' failed
{code}
Any advice ? 

> make install fails the second time
> --
>
> Key: MESOS-6851
> URL: https://issues.apache.org/jira/browse/MESOS-6851
> Project: Mesos
>  Issue Type: Bug
>  Components: build
>Reporter: James Peach
>Priority: Major
>
> Run {{make install}} twice and the second time will fail when it tries to 
> overwrite symlinks:
> {code}
> make[4]: Entering directory '/home/jpeach/upstream/mesos/build/src'
> cd //opt/mesos/etc/mesos && \
>   ln -s mesos-agent-env.sh.template mesos-slave-env.sh.template
> ln: failed to create symbolic link 'mesos-slave-env.sh.template': File exists
> Makefile:12952: recipe for target 'copy-template-and-create-symlink' failed
> make[4]: *** [copy-template-and-create-symlink] Error 1
> make[4]: Leaving directory '/home/jpeach/upstream/mesos/build/src'
> Makefile:12487: recipe for target 'install-data-am' failed
> make[3]: *** [install-data-am] Error 2
> make[3]: Leaving directory '/home/jpeach/upstream/mesos/build/src'
> Makefile:12197: recipe for target 'install-am' failed
> make[2]: *** [install-am] Error 2
> make[2]: Leaving directory '/home/jpeach/upstream/mesos/build/src'
> Makefile:12191: recipe for target 'install' failed
> make[1]: *** [install] Error 2
> make[1]: Leaving directory '/home/jpeach/upstream/mesos/build/src'
> Makefile:764: recipe for target 'install-recursive' failed
> make: *** [install-recursive] Error 1
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (MESOS-6851) make install fails the second time

[Jorge Machado (JIRA)]

[ 
https://issues.apache.org/jira/browse/MESOS-6851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808527#comment-16808527
 ] 

Jorge Machado commented on MESOS-6851:
--

So I try it right now and it does not fit. It would much easier to provide deb 
and rpms from the project... 

My commands on Jenkins on a container ubuntu:1804: 
{code:java}
sh("git clone https://github.com/mesosphere/mesos-deb-packaging.git | true ")
dir("mesos-deb-packaging"){
echo("Uninstalling existing installation")
sh("cd mesos-repo/build && make uninstall | true")
echo("The next step is going to take 5 min for cloning the repo")
sh("./build_mesos --repo https://github.com/apache/mesos?tag=${MESOS_VERSION} 
--configure-flags '--enable-ssl --enable-libevent --disable-python 
--disable-java'")
}{code}
Logs: 
{code:java}
make  install-exec-hook

make[4]: Entering directory 
'/var/lib/jenkins/workspace/user/machjor/build_rpm_mesos_machjor@2/mesos-deb-packaging/mesos-repo/build/src'
cd 
/var/lib/jenkins/workspace/user/machjor/build_rpm_mesos_machjor@2/mesos-deb-packaging/mesos-repo/toor//usr/etc/mesos
 && \
  ln -s mesos-agent-env.sh.template mesos-slave-env.sh.template
ln: failed to create symbolic link 'mesos-slave-env.sh.template': File exists
Makefile:14819: recipe for target 'copy-template-and-create-symlink' failed
make[4]: *** [copy-template-and-create-symlink] Error 1
make[4]: Leaving directory 
'/var/lib/jenkins/workspace/user/machjor/build_rpm_mesos_machjor@2/mesos-deb-packaging/mesos-repo/build/src'
Makefile:14338: recipe for target 'install-data-am' failed
{code}
Any advice ? 

> make install fails the second time
> --
>
> Key: MESOS-6851
> URL: https://issues.apache.org/jira/browse/MESOS-6851
> Project: Mesos
>  Issue Type: Bug
>  Components: build
>Reporter: James Peach
>Priority: Major
>
> Run {{make install}} twice and the second time will fail when it tries to 
> overwrite symlinks:
> {code}
> make[4]: Entering directory '/home/jpeach/upstream/mesos/build/src'
> cd //opt/mesos/etc/mesos && \
>   ln -s mesos-agent-env.sh.template mesos-slave-env.sh.template
> ln: failed to create symbolic link 'mesos-slave-env.sh.template': File exists
> Makefile:12952: recipe for target 'copy-template-and-create-symlink' failed
> make[4]: *** [copy-template-and-create-symlink] Error 1
> make[4]: Leaving directory '/home/jpeach/upstream/mesos/build/src'
> Makefile:12487: recipe for target 'install-data-am' failed
> make[3]: *** [install-data-am] Error 2
> make[3]: Leaving directory '/home/jpeach/upstream/mesos/build/src'
> Makefile:12197: recipe for target 'install-am' failed
> make[2]: *** [install-am] Error 2
> make[2]: Leaving directory '/home/jpeach/upstream/mesos/build/src'
> Makefile:12191: recipe for target 'install' failed
> make[1]: *** [install] Error 2
> make[1]: Leaving directory '/home/jpeach/upstream/mesos/build/src'
> Makefile:764: recipe for target 'install-recursive' failed
> make: *** [install-recursive] Error 1
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (MESOS-6851) make install fails the second time

[Jorge Machado (JIRA)]

[ 
https://issues.apache.org/jira/browse/MESOS-6851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16808477#comment-16808477
 ] 

Jorge Machado commented on MESOS-6851:
--

same problem here. I just checked for master and I cannot find the LN_S . Where 
did you fixed it [~frankscholten] ? can you make a PR ?

> make install fails the second time
> --
>
> Key: MESOS-6851
> URL: https://issues.apache.org/jira/browse/MESOS-6851
> Project: Mesos
>  Issue Type: Bug
>  Components: build
>Reporter: James Peach
>Priority: Major
>
> Run {{make install}} twice and the second time will fail when it tries to 
> overwrite symlinks:
> {code}
> make[4]: Entering directory '/home/jpeach/upstream/mesos/build/src'
> cd //opt/mesos/etc/mesos && \
>   ln -s mesos-agent-env.sh.template mesos-slave-env.sh.template
> ln: failed to create symbolic link 'mesos-slave-env.sh.template': File exists
> Makefile:12952: recipe for target 'copy-template-and-create-symlink' failed
> make[4]: *** [copy-template-and-create-symlink] Error 1
> make[4]: Leaving directory '/home/jpeach/upstream/mesos/build/src'
> Makefile:12487: recipe for target 'install-data-am' failed
> make[3]: *** [install-data-am] Error 2
> make[3]: Leaving directory '/home/jpeach/upstream/mesos/build/src'
> Makefile:12197: recipe for target 'install-am' failed
> make[2]: *** [install-am] Error 2
> make[2]: Leaving directory '/home/jpeach/upstream/mesos/build/src'
> Makefile:12191: recipe for target 'install' failed
> make[1]: *** [install] Error 2
> make[1]: Leaving directory '/home/jpeach/upstream/mesos/build/src'
> Makefile:764: recipe for target 'install-recursive' failed
> make: *** [install-recursive] Error 1
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (SPARK-27208) RestSubmissionClient only supports http

[Jorge Machado (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-27208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806495#comment-16806495
 ] 

Jorge Machado commented on SPARK-27208:
---

Any help here please ? I really think this is broken . 

> RestSubmissionClient only supports http
> ---
>
> Key: SPARK-27208
> URL: https://issues.apache.org/jira/browse/SPARK-27208
> Project: Spark
>  Issue Type: Bug
>  Components: Mesos
>Affects Versions: 2.4.0
>    Reporter: Jorge Machado
>Priority: Minor
>
> As stand of now the class RestSubmissionClient does not support https, which 
> fails for example if we run mesos master with ssl and in cluster mode. 
> The spark-submit command fails with: Mesos cluster mode is only supported 
> through the REST submission API
>  
> I create a PR for this which checks if the master endpoint given can speak 
> ssl before submitting the command. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Re: [MESOS-8248] - Expose information about GPU assigned to a task

[Jorge Machado]

another way would be to just use cadvisor

> On 22 Mar 2019, at 08:35, Jorge Machado  wrote:
> 
> Hi Mesos devs, 
> 
> In our use case from mesos we need to get gpu resource usage per task and 
> build dashboards on grafana for it.  Getting the metrics to Grafana we will 
> send the metrics to prometheus the main problem is how to get the metrics in 
> a reliable way. 
> I proposing the following: 
> 
> Changing the mesos.proto and mesos.proto under v1 and on ResourceStatistics 
> message add: 
> 
> //GPU statistics for each container
> optional int32 gpu_idx = 50;
> optional string gpu_uuid = 51;
> optional string device_name = 52;
> optional uint64 gpu_memory_used_mb = 53;
> optional uint64 gpu_memory_total_mb = 54;
> optional double gpu_usage = 55;
> optional int32 gpu_temperature = 56;
> optional int32 gpu_frequency_MHz = 57;
> optional int32 gpu_power_used_W = 58;
> 
> For starters I would like to change NvidiaGpuIsolatorProcess at isolator.cpp 
> and there get the nvml call for the usage method. As I’m new to this I need 
> some guidelines please. 
> 
> My questions:  
> 
> Does the NvidiaGpuIsolatorProcess runs already inside the container or just 
> outside in the agent ? (I’m assuming outside)
> From what I saw on the cpu metrics they are gathered inside the container for 
> the gpu we could do it in the NvidiaGpuIsolatorProcess and get the metrics 
> via the host. 
> Anything more that I should check ? 
> 
> Thanks a lot
> 
> Jorge Machado
> www.jmachado.me
> 
> 
> 
> 
> 

[MESOS-8248] - Expose information about GPU assigned to a task

[Jorge Machado]

Hi Mesos devs, 

In our use case from mesos we need to get gpu resource usage per task and build 
dashboards on grafana for it.  Getting the metrics to Grafana we will send the 
metrics to prometheus the main problem is how to get the metrics in a reliable 
way. 
I proposing the following: 

Changing the mesos.proto and mesos.proto under v1 and on ResourceStatistics 
message add: 

//GPU statistics for each container
optional int32 gpu_idx = 50;
optional string gpu_uuid = 51;
optional string device_name = 52;
optional uint64 gpu_memory_used_mb = 53;
optional uint64 gpu_memory_total_mb = 54;
optional double gpu_usage = 55;
optional int32 gpu_temperature = 56;
optional int32 gpu_frequency_MHz = 57;
optional int32 gpu_power_used_W = 58;

For starters I would like to change NvidiaGpuIsolatorProcess at isolator.cpp 
and there get the nvml call for the usage method. As I’m new to this I need 
some guidelines please. 

My questions:  

Does the NvidiaGpuIsolatorProcess runs already inside the container or just 
outside in the agent ? (I’m assuming outside)
From what I saw on the cpu metrics they are gathered inside the container for 
the gpu we could do it in the NvidiaGpuIsolatorProcess and get the metrics via 
the host. 
Anything more that I should check ? 

Thanks a lot

Jorge Machado
www.jmachado.me