Re: [VOTE] Standardize Spark Exception Messages SPIP

[EveLiao]

+1 
Thanks!



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: get method guid prefix for file parts for write

[EveLiao]

If I understand your problem correctly, the prefix you provided is actually
"-" + UUID. You can get it by uuid generator like
https://docs.python.org/3/library/uuid.html#uuid.uuid4.



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Test Failures on macOS 10.15.4

[EveLiao]

Hi,

I am new in the development of Spark. When I tried to run unit tests locally
on macOS 10.15.4, everything went smoothly except a single testcase -
SPARK-6330 regression test. After a few hours struggling with it, I moved to
Linux and it passed magically. My OS is Ubuntu 18.0.4. 

Digging into the code, I believe the intention of the test is to validate
that the distributed filesystem's schema is interpreted from the file path
if no default filesystem provided. And it should avoid the exception: 
"IllegalArgumentException: Wrong FS: hdfs://..., expected: file:///".
Instead, the code goes further and meets errors like "UnknownHostException"
when connecting to the remote system as it is a fake file path. However, the
test on my local environment broke because it throws another exception: 
"java.lang.IllegalArgumentException: Pathname  from hdfs://nonexistent is
not a valid DFS filename." when connecting to the remote.

The code is the below:

  test("SPARK-6330 regression test") {
// In 1.3.0, save to fs other than file: without configuring
core-site.xml would get:
// IllegalArgumentException: Wrong FS: hdfs://..., expected: file:///
intercept[Throwable] {
  spark.read.parquet("file:///nonexistent")
}
val errorMessage = intercept[Throwable] {
  spark.read.parquet("hdfs://nonexistent")
}.toString
assert(errorMessage.contains("UnknownHostException"))
  }

I am wondering if anyone has seen the same broken test before. If so, what
tweaks did you do to make it pass? It could be something I missed when
setting up my local environment. I was using Hadoop3.2 and Hive2.3.

If it is due to the discrepancy of OS systems, does it make sense to make a
change to the test case to help local development? Though we have Jenkins,
we may still need to run tests locally sometimes. My proposals would be:

1. assert(!errorMessage.contains("Wrong FS"))
The risk is later version of Hadoop might change the content of the error
message.

2. assert(errorMessage.contains("UnknownHostException") ||
errorMessage.contains("not a valid DFS filename"))

Any suggestions would be really appreciated. Thanks for your time!



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Contributing to JIRA Maintenance

[EveLiao]

Sure. That's wonderful. My JIRA account is @EveLiao.

I will make a list of issues that I consider to be okay to close. And let
you know when we can make a call.

Thanks!



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Re: Contributing to JIRA Maintenance

[EveLiao]

Hi,

I am a novice in Spark and would love to help with JIRA Maintenance as a
start to contribution.

However, how can I edit the status of the issue in JIRA? It looks like that
I don't have access to it.
Should I comment below the issue and @ an administrator?

For example, I think the below issue is actually an intended behavior
decided via an discussion of contributers. Instead, the user should change
the way how they use it. So I would like to mark it as "invalid" if you
think it makes sense.

https://issues.apache.org/jira/browse/SPARK-21774

However, I could not find a button to edit its status. The page I looked at
is like the below:


 

Can you help with that? 

Thanks!



--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org