[jira] [Commented] (FLINK-4316) Make flink-core independent of Hadoop

2016-08-10 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-4316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15415039#comment-15415039
 ] 

ASF GitHub Bot commented on FLINK-4316:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2338


> Make flink-core independent of Hadoop
> -
>
> Key: FLINK-4316
> URL: https://issues.apache.org/jira/browse/FLINK-4316
> Project: Flink
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 1.1.0
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 1.2.0
>
>
> We want to gradually reduce the hard and heavy mandatory dependencies in 
> Hadoop. Hadoop will still be part of (most) flink downloads, but the API 
> projects should not have a hard dependency on Hadoop.
> I suggest to start with {{flink-core}}, because it only depends on Hadoop for 
> the {{Writable}} type, to support seamless operation of Hadoop types.
> I propose to move all {{WritableTypeInfo}}-related classes to the 
> {{flink-hadoop-compatibility}} project and access them via reflection in the 
> {{TypeExtractor}}.
> That way, {{Writable}} types will be out of the box supported if users have 
> the {{flink-hadoop-compatibility}} project in the classpath.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4316) Make flink-core independent of Hadoop

2016-08-09 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-4316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15413275#comment-15413275
 ] 

ASF GitHub Bot commented on FLINK-4316:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2338
  
Good point. I'll fix those tests...


> Make flink-core independent of Hadoop
> -
>
> Key: FLINK-4316
> URL: https://issues.apache.org/jira/browse/FLINK-4316
> Project: Flink
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 1.1.0
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 1.2.0
>
>
> We want to gradually reduce the hard and heavy mandatory dependencies in 
> Hadoop. Hadoop will still be part of (most) flink downloads, but the API 
> projects should not have a hard dependency on Hadoop.
> I suggest to start with {{flink-core}}, because it only depends on Hadoop for 
> the {{Writable}} type, to support seamless operation of Hadoop types.
> I propose to move all {{WritableTypeInfo}}-related classes to the 
> {{flink-hadoop-compatibility}} project and access them via reflection in the 
> {{TypeExtractor}}.
> That way, {{Writable}} types will be out of the box supported if users have 
> the {{flink-hadoop-compatibility}} project in the classpath.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4316) Make flink-core independent of Hadoop

2016-08-09 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-4316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15413259#comment-15413259
 ] 

ASF GitHub Bot commented on FLINK-4316:
---

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/2338
  
+1 to merge, once the failing tests are fixed.

I think the exception to the API stability check is okay in this case. The 
class is still in the same java package.

This are the test failures:
```
Results :

Failed tests: 
  PojoTypeExtractionTest.testPojoWC:203->checkWCPojoAsserts:244 position of 
field complex.valueType wrong expected:<2> but was:<5>
Tests in error: 
  TypeInfoParserTest.testMultiDimensionalArray:321 » IllegalArgument 
String coul...
  TypeInfoParserTest.testPojoType:190 » IllegalArgument String could not 
be pars...
```


> Make flink-core independent of Hadoop
> -
>
> Key: FLINK-4316
> URL: https://issues.apache.org/jira/browse/FLINK-4316
> Project: Flink
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 1.1.0
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 1.2.0
>
>
> We want to gradually reduce the hard and heavy mandatory dependencies in 
> Hadoop. Hadoop will still be part of (most) flink downloads, but the API 
> projects should not have a hard dependency on Hadoop.
> I suggest to start with {{flink-core}}, because it only depends on Hadoop for 
> the {{Writable}} type, to support seamless operation of Hadoop types.
> I propose to move all {{WritableTypeInfo}}-related classes to the 
> {{flink-hadoop-compatibility}} project and access them via reflection in the 
> {{TypeExtractor}}.
> That way, {{Writable}} types will be out of the box supported if users have 
> the {{flink-hadoop-compatibility}} project in the classpath.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4316) Make flink-core independent of Hadoop

2016-08-05 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-4316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15409493#comment-15409493
 ] 

ASF GitHub Bot commented on FLINK-4316:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/2338
  
@StephanEwen that's exactly what I was just going to suggest for Scala.  


> Make flink-core independent of Hadoop
> -
>
> Key: FLINK-4316
> URL: https://issues.apache.org/jira/browse/FLINK-4316
> Project: Flink
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 1.1.0
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 1.2.0
>
>
> We want to gradually reduce the hard and heavy mandatory dependencies in 
> Hadoop. Hadoop will still be part of (most) flink downloads, but the API 
> projects should not have a hard dependency on Hadoop.
> I suggest to start with {{flink-core}}, because it only depends on Hadoop for 
> the {{Writable}} type, to support seamless operation of Hadoop types.
> I propose to move all {{WritableTypeInfo}}-related classes to the 
> {{flink-hadoop-compatibility}} project and access them via reflection in the 
> {{TypeExtractor}}.
> That way, {{Writable}} types will be out of the box supported if users have 
> the {{flink-hadoop-compatibility}} project in the classpath.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4316) Make flink-core independent of Hadoop

2016-08-05 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-4316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15409435#comment-15409435
 ] 

ASF GitHub Bot commented on FLINK-4316:
---

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/2338
  
The Scala type extraction has now also been addressed.


> Make flink-core independent of Hadoop
> -
>
> Key: FLINK-4316
> URL: https://issues.apache.org/jira/browse/FLINK-4316
> Project: Flink
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 1.1.0
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 1.2.0
>
>
> We want to gradually reduce the hard and heavy mandatory dependencies in 
> Hadoop. Hadoop will still be part of (most) flink downloads, but the API 
> projects should not have a hard dependency on Hadoop.
> I suggest to start with {{flink-core}}, because it only depends on Hadoop for 
> the {{Writable}} type, to support seamless operation of Hadoop types.
> I propose to move all {{WritableTypeInfo}}-related classes to the 
> {{flink-hadoop-compatibility}} project and access them via reflection in the 
> {{TypeExtractor}}.
> That way, {{Writable}} types will be out of the box supported if users have 
> the {{flink-hadoop-compatibility}} project in the classpath.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4316) Make flink-core independent of Hadoop

2016-08-05 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-4316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15409391#comment-15409391
 ] 

ASF GitHub Bot commented on FLINK-4316:
---

GitHub user StephanEwen opened a pull request:

https://github.com/apache/flink/pull/2338

[FLINK-4316] [core] [hadoop compatibility] Make flink-core independent of 
Hadoop

This pull request moves all 'Writable' related code to the 
'flink-hadoop-compatibility' project
and uses reflection in 'flink-core' to instantiate WritableTypeInfo when 
needed.

The test are also moved and adjusted to reflect this.

One remaining issue is how to instantiate the WritableTypeInfo from within 
the Scala Macros. I'd need @aljoscha or @tillrohrmann to look at this.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/StephanEwen/incubator-flink core_no_hadoop

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2338.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2338


commit dc509d6d4b5388a40cdd13f7102daf4a7c932dba
Author: Stephan Ewen 
Date:   2016-08-05T10:56:41Z

[hadoop compatibility] Remove unneeded dependency to 'flink-clients'

Also includes minor code cleanups for warnings and more explicit 
serialization behavior.

commit 538759b047b337c3f2199ac400790f9a15ed4200
Author: Stephan Ewen 
Date:   2016-08-05T12:27:48Z

[FLINK-4316] [core] [hadoop compatibility] Make flink-core independent of 
Hadoop

This commit moves all 'Writable' related code to the 
'flink-hadoop-compatibility' project
and uses reflection in 'flink-core' to instantiate WritableTypeInfo when 
needed.




> Make flink-core independent of Hadoop
> -
>
> Key: FLINK-4316
> URL: https://issues.apache.org/jira/browse/FLINK-4316
> Project: Flink
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 1.1.0
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 1.2.0
>
>
> We want to gradually reduce the hard and heavy mandatory dependencies in 
> Hadoop. Hadoop will still be part of (most) flink downloads, but the API 
> projects should not have a hard dependency on Hadoop.
> I suggest to start with {{flink-core}}, because it only depends on Hadoop for 
> the {{Writable}} type, to support seamless operation of Hadoop types.
> I propose to move all {{WritableTypeInfo}}-related classes to the 
> {{flink-hadoop-compatibility}} project and access them via reflection in the 
> {{TypeExtractor}}.
> That way, {{Writable}} types will be out of the box supported if users have 
> the {{flink-hadoop-compatibility}} project in the classpath.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-4316) Make flink-core independent of Hadoop

2016-08-05 Thread Stephan Ewen (JIRA) 

[ 
https://issues.apache.org/jira/browse/FLINK-4316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15409384#comment-15409384
 ] 

Stephan Ewen commented on FLINK-4316:
-

An alternative would be to copy the Hadoop {{Writable}} type (simple interface) 
into {{link-core}}. We did that before, it works well and is a much easier 
solution (no reflection work), but comes with two issues:
  - We need to carry that class in the flink codebase
  - There will be multiple versions of {{Writable}} and in theory, these could 
lead to class cast exceptions.

> Make flink-core independent of Hadoop
> -
>
> Key: FLINK-4316
> URL: https://issues.apache.org/jira/browse/FLINK-4316
> Project: Flink
>  Issue Type: Bug
>  Components: Core
>Affects Versions: 1.1.0
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
> Fix For: 1.2.0
>
>
> We want to gradually reduce the hard and heavy mandatory dependencies in 
> Hadoop. Hadoop will still be part of (most) flink downloads, but the API 
> projects should not have a hard dependency on Hadoop.
> I suggest to start with {{flink-core}}, because it only depends on Hadoop for 
> the {{Writable}} type, to support seamless operation of Hadoop types.
> I propose to move all {{WritableTypeInfo}}-related classes to the 
> {{flink-hadoop-compatibility}} project and access them via reflection in the 
> {{TypeExtractor}}.
> That way, {{Writable}} types will be out of the box supported if users have 
> the {{flink-hadoop-compatibility}} project in the classpath.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)