Re: [swift-evolution] [Discussion] Swift for Data Science / ML / Big Data analytics

[Brent Royal-Gordon via swift-evolution]

> On Oct 31, 2017, at 9:31 PM, Chris Lattner via swift-evolution 
>  wrote:
> 
> Since you bring it up, Python exceptions will be annoying - As with other 
> languages, Python can throw from an arbitrary expression.  Modeling 
> everything as throws in Swift would be super-annoying and unergonomic for the 
> programmer, because we'd require 'try' everywhere.  Thoughts on what to do 
> about that are welcome!


Add a `throws!` keyword to Swift, indicating that if a call is not covered by a 
`try` or `try?` keyword, a `try!` should be implicitly added.

-- 
Brent Royal-Gordon
Architechies

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Discussion] Swift for Data Science / ML / Big Data analytics

[Chris Lattner via swift-evolution]

> On Nov 2, 2017, at 4:39 AM, Richard Wei  wrote:
>> Question though: how does it work?  Say the first np.array call threw a 
>> python exception:
>> 
>>> try Python.do {
>>> let a = np.array([1, 2, 3])
>>> let b = np.array([[2], [4]])
>>> print(a.dot(b)) // matrix mul with incompatible shapes
>>> }
>> 
>> 
>> We can definitely make the python glue code notice it, catch it and squirrel 
>> it away somewhere, but without compiler hacks we couldn’t make it jump out 
>> of the closure.  This means that np.array would have to return something, 
>> and the calls below it would still execute, or am I missing something?
> 
> We make PythonObjects internally nullable (only in the exception-caught 
> state). The second np.array would just return a null PythonObject. 
> 
> To be specific, we define three states in the python overlay:
> - Normal state: PythonObjects are guaranteed to be non-null. Any exception 
> traps.
> - Exception-catching state: PythonObjects are still guaranteed to be 
> non-null. Any exception triggers the exception-caught state.
> - Exception-caught state: PythonObjects are nullable — all python expressions 
> return a null PythonObject.
> 
> The exception-catching state is entered during the execution of Python.do’s 
> body.


I don’t think that will work well in practice: if you’re intermixing swift and 
python calls, it would be extremely surprising to see values be computed but 
produce no value.  This seems similar to the ObjC "messaging nil” problem.

That said, I’ve been experimenting with another approach that seems to work 
well in practice.  Due to the language mismatch, a Swift programmer is going to 
have to decide whether they care about a throwing call or not.  As such, I 
think it makes sense to make this explicit with some kind of syntax - either a 
postfix operator like ^ or a method like “throwing”.  This allows you to write 
(when we have language sugar) either "a.foo()” or “a.throwing.foo()” if you 
want to handle the case when foo throws an exception.

This sounds yucky, but actually works pretty well in practice because the “try” 
warnings (about the absence/excess of try) dovetail really well.  Without 
language support we get something like this:

  // import pickle
  let pickle = Python.import("pickle")

  // file = open(filename)
  guard let file = try? Python.open.throwing.call(args: filename) else {
fatalError("""
   Didn't find data file at \(filename)!
   Update the DataPath at the top of the file.\n
   """)
  }

  // blob = file.read()
  let blob = file.get(member: "read").call(args: [])

  // a, b, c = pickle.loads(blob)
  let (a, b, c) = pickle.get(member: "loads").call(args: blob).get3Tuple()


When we allow sugaring 'x.get(member: “foo”)’ into ‘x.foo’ and allow sugaring 
x.call(args: a, b, c) into ‘x(a, b,c)’, we’ll get this code:

  // import pickle
  let pickle = Python.import("pickle")

  // file = open(filename)
  guard let file = try? Python.open.throwing(filename) else {
fatalError("""
   Didn't find data file at \(filename)!
   Update the DataPath at the top of the file.\n
   """)
  }

  // blob = file.read()
  let blob = file.read()

  // a, b, c = pickle.loads(blob)
  let (a, b, c) = pickleloads(blob).get3Tuple()


Which is pretty nice.   We can talk about making tuple destructuring extensible 
later.  :-)


Here’s the prototype that backs the code above - it is hacky, has known 
suboptimality, and probably isn’t self contained.  I’ll clean this up and write 
up a proposal for the two bits of sugar above when I have time.

-Chris




/// This represents the result of a failable operation when working with
/// python values.
public enum PythonError : Error {
  /// This represents a null IUO being passed into a PyRef.  This should only
  /// occur when working with C APIs.
  case nullValue
  /// This represents an exception thrown out of a Python API.  This can occur
  /// on calls.
  case exception(_ : PyRef)
}

/// Reference to a Python value.  This is always non-null and always owning of
/// the underlying value.
public final class PyRef : PythonObjectionable {
  private var state : UnsafeMutablePointer

  var borrowedPyObject : UnsafeMutablePointer {
return state
  }
  var ownedPyObject : UnsafeMutablePointer {
return py_INCREF(state)
  }
  public init(ownedThrowing: UnsafeMutablePointer!) throws {
if let owned = ownedThrowing {
  state = owned
} else {
  throw PythonError.nullValue
}
  }
  public convenience
  init(borrowedThrowing: UnsafeMutablePointer!) throws {
try self.init(ownedThrowing: borrowedThrowing)
py_INCREF(state)
  }
  deinit {
py_DECREF(state)
  }

  public convenience init(owned: UnsafeMutablePointer!) {
try! self.init(ownedThrowing: owned)
  }
  public convenience init(borrowed: UnsafeMutablePointer!) {
try! self.init(borrowedThrowing: borrowed)
  }
  public convenience init?(python: UnsafeMutablePointer) {
self.init(bo

Re: [swift-evolution] [Discussion] Swift for Data Science / ML / Big Data analytics

[Chris Lattner via swift-evolution]

> On Nov 2, 2017, at 1:58 PM, Rocky Wei via swift-evolution 
>  wrote:
> 
> Hi Chris and everyone else,
> 
> So PerfectlySoft made Perfect-Python months ago to help Swift import python 
> objects and libraries. However, it may reduce the strong type checking. Any 
> idea to avoid it?
> 
> https://github.com/PerfectlySoft/Perfect-Python 
> 
Cool, I wasn’t aware of this.  It looks like a straight-forward wrapper for the 
Python C API.  

Here are some random questions:

why do you make the conversions from Python types to Swift types non-failable?  
This init I’d expect to be failable, for example:
https://github.com/PerfectlySoft/Perfect-Python/blob/master/Sources/PerfectPython/PerfectPython.swift#L34
 



I understand that you’re wrapping PyObject* with a class to get ARC behavior, 
but why make it “open”?  What does subclassability mean for your PyObj type?
https://github.com/PerfectlySoft/Perfect-Python/blob/master/Sources/PerfectPython/PerfectPython.swift#L183


Why do you print errors when you throw, instead of including the details in the 
error that gets thrown?
https://github.com/PerfectlySoft/Perfect-Python/blob/master/Sources/PerfectPython/PerfectPython.swift#L223
 


Why include a fixed list of supported types, instead of using a protocol to 
make it extensible?
https://github.com/PerfectlySoft/Perfect-Python/blob/master/Sources/PerfectPython/PerfectPython.swift#L260
 


What’s this defer doing?
https://github.com/PerfectlySoft/Perfect-Python/blob/master/Sources/PerfectPython/PerfectPython.swift#L423
 


-Chris

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[swift-evolution] [Discussion] Swift for Data Science / ML / Big Data analytics

[Rocky Wei via swift-evolution]

Hi Chris and everyone else,

So PerfectlySoft made Perfect-Python months ago to help Swift import python 
objects and libraries. However, it may reduce the strong type checking. Any 
idea to avoid it?

https://github.com/PerfectlySoft/Perfect-Python 


You can also try Perfect-TensorFlow as well - the TensorFlow Swift Binding, 
which supports latest one -  TensorFlow 1.4.0 released today.

https://github.com/PerfectlySoft/Perfect-TensorFlow 


Rocky 

> On Oct 28, 2017, at 9:45 AM, Maxim Veksler via swift-evolution 
>  > wrote:
> > 
> > Hey Guys,
> > 
> > The big data and machine learning world is dominated by Python, Scala an R. 
> > 
> > I'm a Swifter by heart, but not so much by tools of trait. 
> 
> Hi Max,
> 
> I’m very interested in this topic, with a specific focus on Python.  It isn’t 
> the immediate thing on my priority list to deal with, but I hope that we get 
> to push on this.
> 
> In short, I think we should build a simple Swift/Python interop story.  This 
> sort of thing has be built numerous times for many languages (owing to 
> Python’s great support for embed-ability), including things like PyObjC, 
> boost.python, and many others.
> 
> In Swift, it is straightforward to make this example 
> (http://cs231n.github.io/python-numpy-tutorial/#numpy-arrays 
>  
>  >) look 
> something like this:
> 
>   let np = Python.import(“numpy”)   // Returns a value of type 
> Python.Object.
>   let a = np.array([1, 2, 3])
>   print(type(a))// Whether we want to support type(x) or use the 
> Swift equivalent would be up for discussion of course!
>   print(a.shape)
>   print(a[0], a[1], a[2])
>   a[0] = 5
>   print(a)
> 
>   let b = np.array([[1,2,3],[4,5,6]])
>   print(b.shape)
>   print(b[0, 0], b[0, 1], b[1, 0])
> 
> … which is to say, exactly identical to the Python version except that new 
> variables need to be declared with let/var.  This can be done by blessing 
> Python.Object (which is identical to “PyObject*” at the machine level) with 
> some special dynamic name lookup behavior:  Dot syntax turns into a call to 
> PyObject_GetAttrString, subscripts turn into PyObject_GetItem, calls turn 
> into PyObject_Call, etc.  ARC would be implemented with INCREF etc.
> 
> If we do this, the vast majority of the Python ecosystem should be directly 
> usable from within Swift code, and the only a few major syntactic differences 
> (e.g. ranges work differently).  We would add failable inits to the primitive 
> datatypes like Int/String/etc to convert Python.Object values into them, and 
> add the corresponding non-failable conversions from Python.Object to those 
> primitives.
> 
> Overall, I think it will provide a really nice experience, and allow us to 
> leverage the vast majority of the Python ecosystem directly in Swift code. 
> This project would also have much more narrow impact on the Swift compiler 
> than the ObjC importer (since it works completely differently).  For a first 
> cut, I don’t think we would have to worry about Swift classes subclassing 
> Python classes, for example.
> 
> -Chris
> 
> 
> 
> 
> 
> > 
> > I'd appreciate a constructive discussion on how that could be changed.
> > 
> > While R is a non goal for obvious reasons, i'd argue that since both Scala 
> > and Python are general purpose languages, taking them head to head might be 
> > a low hanging fruit.
> > 
> > To make the claim I'd like to reference to projects such as 
> > 
> >  - Hadoop, Spark, Hive are all huge eco-systems which are entirely JVM 
> > based.
> >  - Apache Parquet, a highly efficient column based storage format for big 
> > data analytics which was implemented in Java, and C++.
> >  - Apache Arrow, a physical memory spec that big data systems can use to 
> > allow zero transformations on data transferred between systems. Which (for 
> > obvious reasons) focused on JVM, to C interoperability.
> > 
> > Python's Buffer Protocol which ensures it's predominance (for the time 
> > being) as a prime candidate for data science related projects 
> > https://jeffknupp.com/blog/2017/09/15/python-is-the-fastest-growing-programming-language-due-to-a-feature-youve-never-heard-of/
> >  
> > 
> >  
> >  >  
> > >
> > 
> > While Swift's Memory Ownership man

Re: [swift-evolution] [Discussion] Swift for Data Science / ML / Big Data analytics

[Richard Wei via swift-evolution]

> On Nov 1, 2017, at 21:13, Chris Lattner via swift-evolution 
>  wrote:
> 
>> On Nov 1, 2017, at 3:20 AM, Richard Wei > > wrote:
 
 Since you bring it up, Python exceptions will be annoying - As with other 
 languages, Python can throw from an arbitrary expression.  Modeling 
 everything as throws in Swift would be super-annoying and unergonomic for 
 the programmer, because we'd require 'try' everywhere.  Thoughts on what 
 to do about that are welcome!
>>> 
>>> Requiring ‘try’ on every statement is annoying, but not having the ability 
>>> to catch python exceptions is annoying too. We could probably make python 
>>> exception handling an opt-in feature. For example:
>>> 
>>> try Python.do {
>>> let a = np.array([1, 2, 3])
>>> let b = np.array([[2], [4]])
>>> print(a.dot(b)) // matrix mul with incompatible shapes
>>> }
>>> catch let error as PythonException {
>>> // Handle PythonError.valueError(“objects are not aligned”)
>>> }
>> 
>> To correct my example:
>> 
>> do {
>> try Python.do {
>> let a = np.array([1, 2, 3])
>> let b = np.array([[2], [4]])
>> print(a.dot(b)) // matrix mul with incompatible shapes
>> }
>> }
>> catch let error as PythonException {
>> // Handle PythonError.valueError(“objects are not aligned”)
>> }
>> 
>> Maybe ‘Python.do {}’ should be called something like ‘Python.safely {}’.
> 
> That’s a super interesting way to model this.  I’ll need to ponder on it 
> more, but  it is certainly a nice ergonomic solution.
> 
> Question though: how does it work?  Say the first np.array call threw a 
> python exception:
> 
>> try Python.do {
>> let a = np.array([1, 2, 3])
>> let b = np.array([[2], [4]])
>> print(a.dot(b)) // matrix mul with incompatible shapes
>> }
> 
> 
> We can definitely make the python glue code notice it, catch it and squirrel 
> it away somewhere, but without compiler hacks we couldn’t make it jump out of 
> the closure.  This means that np.array would have to return something, and 
> the calls below it would still execute, or am I missing something?

We make PythonObjects internally nullable (only in the exception-caught state). 
The second np.array would just return a null PythonObject.

To be specific, we define three states in the python overlay:
- Normal state: PythonObjects are guaranteed to be non-null. Any exception 
traps.
- Exception-catching state: PythonObjects are still guaranteed to be non-null. 
Any exception triggers the exception-caught state.
- Exception-caught state: PythonObjects are nullable — all python expressions 
return a null PythonObject.

The exception-catching state is entered during the execution of Python.do’s 
body.

-Richard

> 
> -Chris
> 
> 
> ___
> swift-evolution mailing list
> swift-evolution@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution



signature.asc
Description: Message signed with OpenPGP
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Discussion] Swift for Data Science / ML / Big Data analytics

[Chris Lattner via swift-evolution]

> On Nov 1, 2017, at 3:20 AM, Richard Wei  wrote:
>>> 
>>> Since you bring it up, Python exceptions will be annoying - As with other 
>>> languages, Python can throw from an arbitrary expression.  Modeling 
>>> everything as throws in Swift would be super-annoying and unergonomic for 
>>> the programmer, because we'd require 'try' everywhere.  Thoughts on what to 
>>> do about that are welcome!
>> 
>> Requiring ‘try’ on every statement is annoying, but not having the ability 
>> to catch python exceptions is annoying too. We could probably make python 
>> exception handling an opt-in feature. For example:
>> 
>> try Python.do {
>> let a = np.array([1, 2, 3])
>> let b = np.array([[2], [4]])
>> print(a.dot(b)) // matrix mul with incompatible shapes
>> }
>> catch let error as PythonException {
>> // Handle PythonError.valueError(“objects are not aligned”)
>> }
> 
> To correct my example: 
> 
> do { 
> try Python.do {
> let a = np.array([1, 2, 3])
> let b = np.array([[2], [4]])
> print(a.dot(b)) // matrix mul with incompatible shapes
> }
> }
> catch let error as PythonException {
> // Handle PythonError.valueError(“objects are not aligned”)
> }
> 
> Maybe ‘Python.do {}’ should be called something like ‘Python.safely {}’.

That’s a super interesting way to model this.  I’ll need to ponder on it more, 
but  it is certainly a nice ergonomic solution.

Question though: how does it work?  Say the first np.array call threw a python 
exception:

> try Python.do {
> let a = np.array([1, 2, 3])
> let b = np.array([[2], [4]])
> print(a.dot(b)) // matrix mul with incompatible shapes
> }


We can definitely make the python glue code notice it, catch it and squirrel it 
away somewhere, but without compiler hacks we couldn’t make it jump out of the 
closure.  This means that np.array would have to return something, and the 
calls below it would still execute, or am I missing something?

-Chris


___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Discussion] Swift for Data Science / ML / Big Data analytics

[Richard Wei via swift-evolution]

> On Nov 1, 2017, at 03:14, Richard Wei via swift-evolution 
>  wrote:
> 
> 
> 
>> On Oct 31, 2017, at 21:31, Chris Lattner via swift-evolution 
>> mailto:swift-evolution@swift.org>> wrote:
>> 
>> Also, for sake of discussion, we’d have to figure out what the type of 
>> MemberLookupResultType would be for Python.  I can see several choices:
>> 
>> 1) Make it return a  "PythonObject!”
>> 2) Make it strict, returning a PythonObject or trapping.
>> 3) Make PythonObject itself nullable internally and return a null 
>> PythonObject.
>> 
>> #1 matches Python semantics directly, because it allows clients who care to 
>> check, but those who don't can ignore it.  The only problem I anticipate is 
>> that it will break things like:
>> 
>> let x = foo.bar
>> let y = x.thing   // fails, because x implicitly promoted to PythonObject?
>>  
>> #3 is gross and cuts against lots of things in Swift (recall when 
>> UnsafePointer itself was implicitly nullable, lets not go back to those bad 
>> old days).  I think that #2 is the least bad tradeoff.
> 
> I agree, PythonObject-or-die is the right trade-off.
> 
>> 
>> Yes, something like this is what I had in mind, but I think that 
>> functionName should only be required to be StringLiteralConvertible (no need 
>> to actually synthesize a real swift string).
>> 
>> 
>> Since you bring it up, Python exceptions will be annoying - As with other 
>> languages, Python can throw from an arbitrary expression.  Modeling 
>> everything as throws in Swift would be super-annoying and unergonomic for 
>> the programmer, because we'd require 'try' everywhere.  Thoughts on what to 
>> do about that are welcome!
> 
> Requiring ‘try’ on every statement is annoying, but not having the ability to 
> catch python exceptions is annoying too. We could probably make python 
> exception handling an opt-in feature. For example:
> 
> try Python.do {
> let a = np.array([1, 2, 3])
> let b = np.array([[2], [4]])
> print(a.dot(b)) // matrix mul with incompatible shapes
> }
> catch let error as PythonException {
> // Handle PythonError.valueError(“objects are not aligned”)
> }

To correct my example: 

do { 
try Python.do {
let a = np.array([1, 2, 3])
let b = np.array([[2], [4]])
print(a.dot(b)) // matrix mul with incompatible shapes
}
}
catch let error as PythonException {
// Handle PythonError.valueError(“objects are not aligned”)
}

Maybe ‘Python.do {}’ should be called something like ‘Python.safely {}’.

-Richard

> 
> Python.do enables exception handling for statements in the body, declared as
> func `do`(_ body: () throws -> T) throws -> T
> 
> When we execute python-throwing statements inside a Python.do{}, 
> PythonException gets thrown. Otherwise, it traps.
> 
> The ‘Python.do’ function would ask the python overlay to enter an 
> "error-catching" state when executing the body closure. We make PythonObjects 
> internally nullable, but guarantee that they are non-null (or trap) when the 
> overlay is not in the “error-caught” state. When a python exception is thrown 
> in the error-catching state, the overlay enters the error-caught state and 
> propagates null through any python computation in the body closure. After the 
> body is executed, throw that python exception.
> 
> However, if there’s a throwing Swift statement after the throwing python 
> statement in the body, the python exception won’t be caught first… So having 
> the body as a non-throwing closure may be a better idea.
> 
> -Richard
> 
>> 
>>> 
>>> class Dog:
>>> 
>>> def __init__(self, name):
>>> self.name = name
>>> self.tricks = []# creates a new empty list for each dog
>>> 
>>> def add_trick(self, trick):
>>> self.tricks.append(trick)
>>> 
>>> With your don’t-modify-the-compiler approach, how can I create a Dog 
>>> instance and add a trick? I probably need to look up the class by name, 
>>> call __init__ manually, etc.
>>> 
>>>   let dogClass = python_getClassByName(“Dog”) // implemented in the Python 
>>> “overlay’, I guess
>>>   let dog = python_createInstance(dogClass)  // implemented in the Python 
>>> “overlay’, I guess
>>>   dog.__init__(“Brianna”)  // uses CustomCallable’s callMember
>>>   dog.add_trick(“Roll over”)  // uses CustomCallable’s callMember
>> 
>> I-am-not-a-python-expert, but I'd expect this to work:
>> 
>>  let dogModule = Python.import("DogModule")
>>  dogModule.Dog("jckarter").add_trick("SILGenGen”)
>>  let dog = dogModule.Dog(“Brianna”)
>>  dog.add_trick(“Roll over)
>> 
>> or equivalently:
>> 
>>  let Dog = Python.import(“DogModule.Dog")
>>  Dog("jckarter").add_trick("SILGenGen”)
>>  let dog = Dog(“Brianna”)
>>  dog.add_trick(“Roll over)
>> 
>> Seems pretty nice to me, with zero “Swift compiler knowledge of Python” 
>> required.
>> 
>> 
>>> With compiler integration, 
>>> 
>>> class Dog : PythonObject {
>>>   init(_ name: Pythonable)
>>>   func add_trick(_ trick: Pytho

Re: [swift-evolution] [Discussion] Swift for Data Science / ML / Big Data analytics

[Richard Wei via swift-evolution]

> On Oct 31, 2017, at 21:31, Chris Lattner via swift-evolution 
>  wrote:
> 
> Also, for sake of discussion, we’d have to figure out what the type of 
> MemberLookupResultType would be for Python.  I can see several choices:
> 
> 1) Make it return a  "PythonObject!”
> 2) Make it strict, returning a PythonObject or trapping.
> 3) Make PythonObject itself nullable internally and return a null 
> PythonObject.
> 
> #1 matches Python semantics directly, because it allows clients who care to 
> check, but those who don't can ignore it.  The only problem I anticipate is 
> that it will break things like:
> 
> let x = foo.bar
> let y = x.thing   // fails, because x implicitly promoted to PythonObject?
>  
> #3 is gross and cuts against lots of things in Swift (recall when 
> UnsafePointer itself was implicitly nullable, lets not go back to those bad 
> old days).  I think that #2 is the least bad tradeoff.

I agree, PythonObject-or-die is the right trade-off.

> 
> Yes, something like this is what I had in mind, but I think that functionName 
> should only be required to be StringLiteralConvertible (no need to actually 
> synthesize a real swift string).
> 
> 
> Since you bring it up, Python exceptions will be annoying - As with other 
> languages, Python can throw from an arbitrary expression.  Modeling 
> everything as throws in Swift would be super-annoying and unergonomic for the 
> programmer, because we'd require 'try' everywhere.  Thoughts on what to do 
> about that are welcome!

Requiring ‘try’ on every statement is annoying, but not having the ability to 
catch python exceptions is annoying too. We could probably make python 
exception handling an opt-in feature. For example:

try Python.do {
let a = np.array([1, 2, 3])
let b = np.array([[2], [4]])
print(a.dot(b)) // matrix mul with incompatible shapes
}
catch let error as PythonException {
// Handle PythonError.valueError(“objects are not aligned”)
}

Python.do enables exception handling for statements in the body, declared as
func `do`(_ body: () throws -> T) throws -> T

When we execute python-throwing statements inside a Python.do{}, 
PythonException gets thrown. Otherwise, it traps.

The ‘Python.do’ function would ask the python overlay to enter an 
"error-catching" state when executing the body closure. We make PythonObjects 
internally nullable, but guarantee that they are non-null (or trap) when the 
overlay is not in the “error-caught” state. When a python exception is thrown 
in the error-catching state, the overlay enters the error-caught state and 
propagates null through any python computation in the body closure. After the 
body is executed, throw that python exception.

However, if there’s a throwing Swift statement after the throwing python 
statement in the body, the python exception won’t be caught first… So having 
the body as a non-throwing closure may be a better idea.

-Richard

> 
>> 
>> class Dog:
>> 
>> def __init__(self, name):
>> self.name = name
>> self.tricks = []# creates a new empty list for each dog
>> 
>> def add_trick(self, trick):
>> self.tricks.append(trick)
>> 
>> With your don’t-modify-the-compiler approach, how can I create a Dog 
>> instance and add a trick? I probably need to look up the class by name, call 
>> __init__ manually, etc.
>> 
>>   let dogClass = python_getClassByName(“Dog”) // implemented in the Python 
>> “overlay’, I guess
>>   let dog = python_createInstance(dogClass)  // implemented in the Python 
>> “overlay’, I guess
>>   dog.__init__(“Brianna”)  // uses CustomCallable’s callMember
>>   dog.add_trick(“Roll over”)  // uses CustomCallable’s callMember
> 
> I-am-not-a-python-expert, but I'd expect this to work:
> 
>   let dogModule = Python.import("DogModule")
>   dogModule.Dog("jckarter").add_trick("SILGenGen”)
>   let dog = dogModule.Dog(“Brianna”)
>   dog.add_trick(“Roll over)
> 
> or equivalently:
> 
>   let Dog = Python.import(“DogModule.Dog")
>   Dog("jckarter").add_trick("SILGenGen”)
>   let dog = Dog(“Brianna”)
>   dog.add_trick(“Roll over)
> 
> Seems pretty nice to me, with zero “Swift compiler knowledge of Python” 
> required.
> 
> 
>> With compiler integration, 
>> 
>>  class Dog : PythonObject {
>>init(_ name: Pythonable)
>>func add_trick(_ trick: Pythonable)
>>  }
> 
> Something like this is possible, but would be substantially more work, be 
> substantially more invasive, and would set the precedent that every other 
> dynamic language would get support hacked directly into Swift.  The only 
> reason I can see this being useful is if we wanted to support the optional 
> typing annotations in Python.  While this would be "nice to have", I think 
> the cost/benefit tradeoff involved is totally wrong for Swift. 
> 
>> With either the true “Python importer” solution or this code-generation 
>> solution, you at least get some level of code completion and basic sanity 
>> check

Re: [swift-evolution] [Discussion] Swift for Data Science / ML / Big Data analytics

[Chris Lattner via swift-evolution]

I tried to send a response to this earlier today but it apparently didn’t get 
out.  If it did, then I apologize in advance for the dupe:

> On Oct 30, 2017, at 10:47 PM, Douglas Gregor  wrote:
 // not magic, things like Int, String and many other conform to this. 
 protocol Pythonable {
  init?(_ : PythonObject)
  func toPython() -> PythonObject
 }
>>> 
>>> It’s not magic unless you expect the compiler or runtime to help with 
>>> conversion between Int/String/etc. and PythonObject, as with 
>>> _ObjectiveCBridgeable.
>> 
>> Right, as I said above “not magic”.  The conformances would be manually 
>> implemented in the Python overlay.  This provides a free implicit conversion 
>> from "T -> Pythonable” for the T’s we care about, and a failable init from 
>> Python back to Swift types.
> 
> Note that, under this scheme,
> 
>   let p: Pythonable = 17
>   let i: Int = p as! i
> 
> will work if Int is Pythonable, but not when p comes back from Python. 

Right.  I don't expect users to traffic in the Pythonable type, it would be 
something that is used internally by the Python overlay.  The only time I'd 
expect them to see it is when they want to make one of their types Pythonable 
(which would be rare, but needs to be possible).

>> I can think of two things that could tip the scale of the discussion:
>> 
>> a) The big question is whether we *want* the ability to write custom 
>> rule-of-5 style behavior for structs, or if we want it to only be used in 
>> extreme cases (like bridging interop in this proposal).  If we *want* to 
>> support it someday, then adding proper “safe” support is best (if possible). 
>>  If we don’t *want* people to use it, then making it Unsafe and ugly is a 
>> reasonable way to go.
>> 
>> b) The ownership proposal is likely to add deinit's to structs.  If it also 
>> adds explicit move initializers, then it is probably the right thing to add 
>> copy initializers also (and thus go with approach #2).  That said,  I’m not 
>> sure how the move initializers will be spelled or if that is the likely 
>> direction.  If it won’t add these, then it is probably better to go with 
>> approach #1.  John, what do you think?
>> 
>>> Presumably, binding to Python is going to require some compiler 
>>> effort—defining how it is that Python objects are 
>>> initialized/copied/moved/destroyed seems like a reasonable part of that 
>>> effort.
>> 
>> Actually no.  If we add these three proposals, there is no other python (or 
>> perl, etc…) specific support needed.  It is all implementable in the overlay.
> 
> Support for working with Python objects would be implementable in the 
> overlay, but the result isn’t necessarily ergonomic (e.g., my “as!” case from 
> a Python-generated integer object to Int, shown above). That might be fine! 
> More comments on this below.

Yeah, I don't think that matters.  The preferred way to do this is to use:

  Int(somePythonValue)

which would be a failable conversion.  somePythonValue would be a PythonObject 
(the struct) not "Pythonable"

 // Magic, allows anyobject-like member lookup on a type when lookup 
 otherwise fails.
 protocol DynamicMemberLookupable {
   associatedtype MemberLookupResultType
   func dynamicMemberLookup(_ : String) -> MemberLookupResultType
 }
>>> 
>>> AnyObject lookup looks for an actual declaration on any type anywhere. One 
>>> could extend that mechanism to, say, return all Python methods and assume 
>>> that you can call any Python method with any PythonObject instance. 
>>> AnyObject lookup is fairly unprincipled as a language feature, because 
>>> there’s no natural scope in which to perform name lookup, and involves 
>>> hacks at many levels that don’t always work (e.g., AnyObject lookup… 
>>> sometimes… fails across multiple source files for hard-to-explain reasons). 
>>> You’re taking on that brokenness if you expand AnyObject lookup to another 
>>> ecosystem.
>> 
>> Yeah, sorry, that’s not what I meant:
> 
> (Good)

Also, for sake of discussion, we’d have to figure out what the type of 
MemberLookupResultType would be for Python.  I can see several choices:

1) Make it return a  "PythonObject!”
2) Make it strict, returning a PythonObject or trapping.
3) Make PythonObject itself nullable internally and return a null PythonObject.

#1 matches Python semantics directly, because it allows clients who care to 
check, but those who don't can ignore it.  The only problem I anticipate is 
that it will break things like:

let x = foo.bar
let y = x.thing   // fails, because x implicitly promoted to PythonObject?
 
#3 is gross and cuts against lots of things in Swift (recall when UnsafePointer 
itself was implicitly nullable, lets not go back to those bad old days).  I 
think that #2 is the least bad tradeoff.

>> Right, something like this could definitely work, but keep in mind that the 
>> Swift compiler knows nothing about Python declarations.  
>> 
>> Perhaps the most straight-forwar

Re: [swift-evolution] [Discussion] Swift for Data Science / ML / Big Data analytics

[Chris Lattner via swift-evolution]

On Mon, Oct 30, 2017 at 10:55 PM, John McCall  wrote:

>
> I can think of two things that could tip the scale of the discussion:
>
> a) The big question is whether we *want* the ability to write custom
> rule-of-5 style behavior for structs, or if we want it to only be used in
> extreme cases (like bridging interop in this proposal).  If we *want* to
> support it someday, then adding proper “safe” support is best (if
> possible).  If we don’t *want* people to use it, then making it Unsafe and
> ugly is a reasonable way to go.
>
> b) The ownership proposal is likely to add deinit's to structs.  If it
> also adds explicit move initializers, then it is probably the right thing
> to add copy initializers also (and thus go with approach #2).  That said,
>  I’m not sure how the move initializers will be spelled or if that is the
> likely direction.  If it won’t add these, then it is probably better to go
> with approach #1.  John, what do you think?
>
>
> I was hoping not to have to add explicit move/copy initializers, perhaps
> ever.
>

Can you elaborate more on why?  I'm even more interested in your the
rationale more than the outcome :-)


I would suggest one of two things:
>   - We could add a Builtin type for these types in Swift.  Because of our
> architecture, this is mostly an IRGen task.
>

Yes, this could definitely work.


>   - We could start working on C++ import.  C++ import is a huge task, but
> we don't really need most of it for this.
>

This could work, but it would be unfortunate to push people to having to
write (unsafe) C++ and import it into Swift to achieve simple things like
this.  I'd prefer going the direction of suggestion #1 above and allowing a
"pure swift and explicitly unsafe" solution.

-Chris
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Discussion] Swift for Data Science / ML / Big Data analytics

[John McCall via swift-evolution]

> On Oct 31, 2017, at 12:43 AM, Chris Lattner  wrote:
> 
> JohnMC: question for you below.
> 
> On Oct 30, 2017, at 1:25 PM, Douglas Gregor  > wrote:
>>> 
>>> Thinking about the Perl case makes it clear to me that this should not be 
>>> built into the compiler as a monolithic thing.  Perl supports several 
>>> different types (SV/AV/HV) which represent different concepts (scalars, 
>>> arrays, hashes) so baking it all together into one thing would be the wrong 
>>> way to map it.  In fact, the magic we need is pretty small, and seems 
>>> generally useful for other things. Consider a design like this:
>>> 
>>> 
>>> // not magic, things like Int, String and many other conform to this. 
>>> protocol Pythonable {
>>>  init?(_ : PythonObject)
>>>  func toPython() -> PythonObject
>>> }
>> 
>> It’s not magic unless you expect the compiler or runtime to help with 
>> conversion between Int/String/etc. and PythonObject, as with 
>> _ObjectiveCBridgeable.
> 
> Right, as I said above “not magic”.  The conformances would be manually 
> implemented in the Python overlay.  This provides a free implicit conversion 
> from "T -> Pythonable” for the T’s we care about, and a failable init from 
> Python back to Swift types.
> 
>>> // Not magic.
>>> struct PythonObject : /*protocols below*/ {
>>>   var state : UnsafePointer
>>> 
>>>   subscript(_ : Pythonable…) -> PythonObject {
>>> ...
>>>   }
>>> }
>>> 
>>> // Magic, must be on the struct definition.  
>>> // Could alternatively allow custom copy/move/… ctors like C++.
>>> protocol CustomValueWitnessTable {
>>>  static func init(..)
>>>  static func copy(..)
>>>  static func move(..)
>>>  static func destroy(..)
>>> }
>> 
>> Swift’s implementation model supports this. As a surface-level construct 
>> it’s going to be mired in UnsafeMutablePointers, and it’s not at all clear 
>> to me that we want this level of control there. 
> 
> There are two ways to implement it:
> 1) static func’s like the above, which are implemented as UnsafePointer's
> 2) Proper language syntax akin to the C++ “rule of 5”.
> 
> The pro’s and con’s of the first approach:
> 
> pro) full explicit control over what happens
> pro) no other new language features necessary to implement this.  The second 
> approach would need something like ownership to be in place.
> con) injects another avenue of unsafety (though it would be explicit, so it 
> isn’t that bad).  It isn’t obvious to me that approach #2 can be safe, but I 
> haven’t thought about it enough.
> ???) discourages people from using this protocol because of its explicit 
> unsafety.
> 
> I can think of two things that could tip the scale of the discussion:
> 
> a) The big question is whether we *want* the ability to write custom 
> rule-of-5 style behavior for structs, or if we want it to only be used in 
> extreme cases (like bridging interop in this proposal).  If we *want* to 
> support it someday, then adding proper “safe” support is best (if possible).  
> If we don’t *want* people to use it, then making it Unsafe and ugly is a 
> reasonable way to go.
> 
> b) The ownership proposal is likely to add deinit's to structs.  If it also 
> adds explicit move initializers, then it is probably the right thing to add 
> copy initializers also (and thus go with approach #2).  That said,  I’m not 
> sure how the move initializers will be spelled or if that is the likely 
> direction.  If it won’t add these, then it is probably better to go with 
> approach #1.  John, what do you think?

I was hoping not to have to add explicit move/copy initializers, perhaps ever.  
I would suggest one of two things:
  - We could add a Builtin type for these types in Swift.  Because of our 
architecture, this is mostly an IRGen task.
  - We could start working on C++ import.  C++ import is a huge task, but we 
don't really need most of it for this.

John.

> 
>> Presumably, binding to Python is going to require some compiler 
>> effort—defining how it is that Python objects are 
>> initialized/copied/moved/destroyed seems like a reasonable part of that 
>> effort.
> 
> Actually no.  If we add these three proposals, there is no other python (or 
> perl, etc…) specific support needed.  It is all implementable in the overlay.
> 
>>> // Magic, allows anyobject-like member lookup on a type when lookup 
>>> otherwise fails.
>>> protocol DynamicMemberLookupable {
>>>   associatedtype MemberLookupResultType
>>>   func dynamicMemberLookup(_ : String) -> MemberLookupResultType
>>> }
>> 
>> AnyObject lookup looks for an actual declaration on any type anywhere. One 
>> could extend that mechanism to, say, return all Python methods and assume 
>> that you can call any Python method with any PythonObject instance. 
>> AnyObject lookup is fairly unprincipled as a language feature, because 
>> there’s no natural scope in which to perform name lookup, and involves hacks 
>> at many levels that don’t always work (e.g., AnyObject lookup… sometimes… 
>> f

Re: [swift-evolution] [Discussion] Swift for Data Science / ML / Big Data analytics

[Douglas Gregor via swift-evolution]

> On Oct 30, 2017, at 9:43 PM, Chris Lattner  wrote:
> 
> JohnMC: question for you below.
> 
> On Oct 30, 2017, at 1:25 PM, Douglas Gregor  > wrote:
>>> 
>>> Thinking about the Perl case makes it clear to me that this should not be 
>>> built into the compiler as a monolithic thing.  Perl supports several 
>>> different types (SV/AV/HV) which represent different concepts (scalars, 
>>> arrays, hashes) so baking it all together into one thing would be the wrong 
>>> way to map it.  In fact, the magic we need is pretty small, and seems 
>>> generally useful for other things. Consider a design like this:
>>> 
>>> 
>>> // not magic, things like Int, String and many other conform to this. 
>>> protocol Pythonable {
>>>  init?(_ : PythonObject)
>>>  func toPython() -> PythonObject
>>> }
>> 
>> It’s not magic unless you expect the compiler or runtime to help with 
>> conversion between Int/String/etc. and PythonObject, as with 
>> _ObjectiveCBridgeable.
> 
> Right, as I said above “not magic”.  The conformances would be manually 
> implemented in the Python overlay.  This provides a free implicit conversion 
> from "T -> Pythonable” for the T’s we care about, and a failable init from 
> Python back to Swift types.

Note that, under this scheme,

let p: Pythonable = 17
let i: Int = p as! i

will work if Int is Pythonable, but not when p comes back from Python. 

> 
>>> // Not magic.
>>> struct PythonObject : /*protocols below*/ {
>>>   var state : UnsafePointer
>>> 
>>>   subscript(_ : Pythonable…) -> PythonObject {
>>> ...
>>>   }
>>> }
>>> 
>>> // Magic, must be on the struct definition.  
>>> // Could alternatively allow custom copy/move/… ctors like C++.
>>> protocol CustomValueWitnessTable {
>>>  static func init(..)
>>>  static func copy(..)
>>>  static func move(..)
>>>  static func destroy(..)
>>> }
>> 
>> Swift’s implementation model supports this. As a surface-level construct 
>> it’s going to be mired in UnsafeMutablePointers, and it’s not at all clear 
>> to me that we want this level of control there. 
> 
> There are two ways to implement it:
> 1) static func’s like the above, which are implemented as UnsafePointer's
> 2) Proper language syntax akin to the C++ “rule of 5”.
> 
> The pro’s and con’s of the first approach:
> 
> pro) full explicit control over what happens
> pro) no other new language features necessary to implement this.  The second 
> approach would need something like ownership to be in place.
> con) injects another avenue of unsafety (though it would be explicit, so it 
> isn’t that bad).  It isn’t obvious to me that approach #2 can be safe, but I 
> haven’t thought about it enough.
> ???) discourages people from using this protocol because of its explicit 
> unsafety.

con) much of the UnsafePointer interface is based on the value witness table, 
so one has to step lightly or work with something like 
COpaquePointer/UnsafePointer.

> 
> I can think of two things that could tip the scale of the discussion:
> 
> a) The big question is whether we *want* the ability to write custom 
> rule-of-5 style behavior for structs, or if we want it to only be used in 
> extreme cases (like bridging interop in this proposal).  If we *want* to 
> support it someday, then adding proper “safe” support is best (if possible).  
> If we don’t *want* people to use it, then making it Unsafe and ugly is a 
> reasonable way to go.
> 
> b) The ownership proposal is likely to add deinit's to structs.  If it also 
> adds explicit move initializers, then it is probably the right thing to add 
> copy initializers also (and thus go with approach #2).  That said,  I’m not 
> sure how the move initializers will be spelled or if that is the likely 
> direction.  If it won’t add these, then it is probably better to go with 
> approach #1.  John, what do you think?
> 
>> Presumably, binding to Python is going to require some compiler 
>> effort—defining how it is that Python objects are 
>> initialized/copied/moved/destroyed seems like a reasonable part of that 
>> effort.
> 
> Actually no.  If we add these three proposals, there is no other python (or 
> perl, etc…) specific support needed.  It is all implementable in the overlay.

Support for working with Python objects would be implementable in the overlay, 
but the result isn’t necessarily ergonomic (e.g., my “as!” case from a 
Python-generated integer object to Int, shown above). That might be fine! More 
comments on this below.

> 
>>> // Magic, allows anyobject-like member lookup on a type when lookup 
>>> otherwise fails.
>>> protocol DynamicMemberLookupable {
>>>   associatedtype MemberLookupResultType
>>>   func dynamicMemberLookup(_ : String) -> MemberLookupResultType
>>> }
>> 
>> AnyObject lookup looks for an actual declaration on any type anywhere. One 
>> could extend that mechanism to, say, return all Python methods and assume 
>> that you can call any Python method with any PythonObject instance. 
>> AnyOb

Re: [swift-evolution] [Discussion] Swift for Data Science / ML / Big Data analytics

[Douglas Gregor via swift-evolution]

> On Oct 30, 2017, at 2:54 PM, David Kopecky  wrote:
> 
> To Chris’s use case point, I see Python -> Swift interop as a very common use 
> case in data science / ML for speeding up certain functions/modules by 
> embedding Swift into Python, like you can with, say, ctypes or python 
> extensions in dynamic C libraries. This is very common practice in the Python 
> universe, it just could be a lot better with Swift instead.
> 
> Things like this *sort of* exist with Swift 
> (https://gist.github.com/jiaaro/e111f0f64d0cdb8aca38 
> ), but it’s not really 
> very mature or functional.
> 
> I would LOVE to be able to speed up a lot of python code by implementing 
> swift libraries and importing into python. 

There are a bunch of ways one could export Swift functionality to Python, from 
a library solution like the C++ Boost.Python 
(http://www.boost.org/doc/libs/1_65_1/libs/python/doc/html/index.html 
), to 
code generators like SWIG or, with some future expansion of metadata, directly 
at runtime via reflection facilities. Python’s dynamic nature makes this 
direction easier.

The harder challenge (IMO!) is to get a Python library into Swift, because 
there isn’t a lot of type information in Python, so you have to show the Python 
in the much-more-strongly-typed Swift somehow (and keep it usable).

- Doug

> 
> - David
> 
>> On Oct 30, 2017, at 13:25, Douglas Gregor via swift-evolution 
>> mailto:swift-evolution@swift.org>> wrote:
>> 
>> 
>> 
>>> On Oct 29, 2017, at 1:34 PM, Chris Lattner via swift-evolution 
>>> mailto:swift-evolution@swift.org>> wrote:
>>> 
 
 On Oct 29, 2017, at 8:23 AM, Chris Lattner via swift-evolution 
 mailto:swift-evolution@swift.org>> wrote:
 
 
> On Oct 29, 2017, at 4:04 AM, Lukas Stabe  > wrote:
> 
>> On 28. Oct 2017, at 23:10, Chris Lattner via swift-evolution 
>> mailto:swift-evolution@swift.org>> wrote:
>> 
>> … which is to say, exactly identical to the Python version except that 
>> new variables need to be declared with let/var.  This can be done by 
>> blessing Python.Object (which is identical to “PyObject*” at the machine 
>> level) with some special dynamic name lookup behavior:  Dot syntax turns 
>> into a call to PyObject_GetAttrString, subscripts turn into 
>> PyObject_GetItem, calls turn into PyObject_Call, etc.  ARC would be 
>> implemented with INCREF etc.
> 
> That sounds like a very interesting prospect. Do you think it would make 
> sense to make the language features that facilitate this (dynamic 
> dispatch of method calls, property accesses, subscript and ARC) available 
> to Swift classes annotated in some way, so that interop like this can be 
> implemented as a library without special treatment by the Swift compiler? 
> This could also enable more dynamic DSL like features.
 
 I haven’t explored enough of the design space to be sure, but I’d want to 
 make sure that a library version of this could be done without giving up 
 ergonomics of the result.  If you were interested in being able to interop 
 with other languages that are dynamically typed and reference counted, 
 then something like this could be possible in principle:
>>> 
>>> Thinking about the Perl case makes it clear to me that this should not be 
>>> built into the compiler as a monolithic thing.  Perl supports several 
>>> different types (SV/AV/HV) which represent different concepts (scalars, 
>>> arrays, hashes) so baking it all together into one thing would be the wrong 
>>> way to map it.  In fact, the magic we need is pretty small, and seems 
>>> generally useful for other things. Consider a design like this:
>>> 
>>> 
>>> // not magic, things like Int, String and many other conform to this. 
>>> protocol Pythonable {
>>>  init?(_ : PythonObject)
>>>  func toPython() -> PythonObject
>>> }
>> 
>> It’s not magic unless you expect the compiler or runtime to help with 
>> conversion between Int/String/etc. and PythonObject, as with 
>> _ObjectiveCBridgeable.
>> 
>>> 
>>> // Not magic.
>>> struct PythonObject : /*protocols below*/ {
>>>   var state : UnsafePointer
>>> 
>>>   subscript(_ : Pythonable…) -> PythonObject {
>>> ...
>>>   }
>>> }
>>> 
>>> // Magic, must be on the struct definition.  
>>> // Could alternatively allow custom copy/move/… ctors like C++.
>>> protocol CustomValueWitnessTable {
>>>  static func init(..)
>>>  static func copy(..)
>>>  static func move(..)
>>>  static func destroy(..)
>>> }
>> 
>> Swift’s implementation model supports this. As a surface-level construct 
>> it’s going to be mired in UnsafeMutablePointers, and it’s not at all clear 
>> to me that we want this level of control there. Presumably, binding to 
>> Python is going to require some compiler effort—defining how it is that 

Re: [swift-evolution] [Discussion] Swift for Data Science / ML / Big Data analytics

[Chris Lattner via swift-evolution]

JohnMC: question for you below.

On Oct 30, 2017, at 1:25 PM, Douglas Gregor  wrote:
>> 
>> Thinking about the Perl case makes it clear to me that this should not be 
>> built into the compiler as a monolithic thing.  Perl supports several 
>> different types (SV/AV/HV) which represent different concepts (scalars, 
>> arrays, hashes) so baking it all together into one thing would be the wrong 
>> way to map it.  In fact, the magic we need is pretty small, and seems 
>> generally useful for other things. Consider a design like this:
>> 
>> 
>> // not magic, things like Int, String and many other conform to this. 
>> protocol Pythonable {
>>  init?(_ : PythonObject)
>>  func toPython() -> PythonObject
>> }
> 
> It’s not magic unless you expect the compiler or runtime to help with 
> conversion between Int/String/etc. and PythonObject, as with 
> _ObjectiveCBridgeable.

Right, as I said above “not magic”.  The conformances would be manually 
implemented in the Python overlay.  This provides a free implicit conversion 
from "T -> Pythonable” for the T’s we care about, and a failable init from 
Python back to Swift types.

>> // Not magic.
>> struct PythonObject : /*protocols below*/ {
>>   var state : UnsafePointer
>> 
>>   subscript(_ : Pythonable…) -> PythonObject {
>> ...
>>   }
>> }
>> 
>> // Magic, must be on the struct definition.  
>> // Could alternatively allow custom copy/move/… ctors like C++.
>> protocol CustomValueWitnessTable {
>>  static func init(..)
>>  static func copy(..)
>>  static func move(..)
>>  static func destroy(..)
>> }
> 
> Swift’s implementation model supports this. As a surface-level construct it’s 
> going to be mired in UnsafeMutablePointers, and it’s not at all clear to me 
> that we want this level of control there. 

There are two ways to implement it:
1) static func’s like the above, which are implemented as UnsafePointer's
2) Proper language syntax akin to the C++ “rule of 5”.

The pro’s and con’s of the first approach:

pro) full explicit control over what happens
pro) no other new language features necessary to implement this.  The second 
approach would need something like ownership to be in place.
con) injects another avenue of unsafety (though it would be explicit, so it 
isn’t that bad).  It isn’t obvious to me that approach #2 can be safe, but I 
haven’t thought about it enough.
???) discourages people from using this protocol because of its explicit 
unsafety.

I can think of two things that could tip the scale of the discussion:

a) The big question is whether we *want* the ability to write custom rule-of-5 
style behavior for structs, or if we want it to only be used in extreme cases 
(like bridging interop in this proposal).  If we *want* to support it someday, 
then adding proper “safe” support is best (if possible).  If we don’t *want* 
people to use it, then making it Unsafe and ugly is a reasonable way to go.

b) The ownership proposal is likely to add deinit's to structs.  If it also 
adds explicit move initializers, then it is probably the right thing to add 
copy initializers also (and thus go with approach #2).  That said,  I’m not 
sure how the move initializers will be spelled or if that is the likely 
direction.  If it won’t add these, then it is probably better to go with 
approach #1.  John, what do you think?

> Presumably, binding to Python is going to require some compiler 
> effort—defining how it is that Python objects are 
> initialized/copied/moved/destroyed seems like a reasonable part of that 
> effort.

Actually no.  If we add these three proposals, there is no other python (or 
perl, etc…) specific support needed.  It is all implementable in the overlay.

>> // Magic, allows anyobject-like member lookup on a type when lookup 
>> otherwise fails.
>> protocol DynamicMemberLookupable {
>>   associatedtype MemberLookupResultType
>>   func dynamicMemberLookup(_ : String) -> MemberLookupResultType
>> }
> 
> AnyObject lookup looks for an actual declaration on any type anywhere. One 
> could extend that mechanism to, say, return all Python methods and assume 
> that you can call any Python method with any PythonObject instance. AnyObject 
> lookup is fairly unprincipled as a language feature, because there’s no 
> natural scope in which to perform name lookup, and involves hacks at many 
> levels that don’t always work (e.g., AnyObject lookup… sometimes… fails 
> across multiple source files for hard-to-explain reasons). You’re taking on 
> that brokenness if you expand AnyObject lookup to another ecosystem.

Yeah, sorry, that’s not what I meant:

> Although it doesn’t really seem like AnyObject lookup is the thing you’re 
> asking for here. It seems more like you want dynamicMemberLookup(_:) to 
> capture “self” and the method name, and then be a callable thing as below…

That’s what I meant :-).

A type that implements this magic protocol would never fail name lookup: 
“foo.bar” would always fall back to calling: foo.dynamicMemberLookup(“ba

Re: [swift-evolution] [Discussion] Swift for Data Science / ML / Big Data analytics

[David Kopecky via swift-evolution]

To Chris’s use case point, I see Python -> Swift interop as a very common use 
case in data science / ML for speeding up certain functions/modules by 
embedding Swift into Python, like you can with, say, ctypes or python 
extensions in dynamic C libraries. This is very common practice in the Python 
universe, it just could be a lot better with Swift instead.

Things like this *sort of* exist with Swift 
(https://gist.github.com/jiaaro/e111f0f64d0cdb8aca38 
), but it’s not really 
very mature or functional.

I would LOVE to be able to speed up a lot of python code by implementing swift 
libraries and importing into python. 

- David

> On Oct 30, 2017, at 13:25, Douglas Gregor via swift-evolution 
>  wrote:
> 
> 
> 
>> On Oct 29, 2017, at 1:34 PM, Chris Lattner via swift-evolution 
>> mailto:swift-evolution@swift.org>> wrote:
>> 
>>> 
>>> On Oct 29, 2017, at 8:23 AM, Chris Lattner via swift-evolution 
>>> mailto:swift-evolution@swift.org>> wrote:
>>> 
>>> 
 On Oct 29, 2017, at 4:04 AM, Lukas Stabe >>> > wrote:
 
> On 28. Oct 2017, at 23:10, Chris Lattner via swift-evolution 
> mailto:swift-evolution@swift.org>> wrote:
> 
> … which is to say, exactly identical to the Python version except that 
> new variables need to be declared with let/var.  This can be done by 
> blessing Python.Object (which is identical to “PyObject*” at the machine 
> level) with some special dynamic name lookup behavior:  Dot syntax turns 
> into a call to PyObject_GetAttrString, subscripts turn into 
> PyObject_GetItem, calls turn into PyObject_Call, etc.  ARC would be 
> implemented with INCREF etc.
 
 That sounds like a very interesting prospect. Do you think it would make 
 sense to make the language features that facilitate this (dynamic dispatch 
 of method calls, property accesses, subscript and ARC) available to Swift 
 classes annotated in some way, so that interop like this can be 
 implemented as a library without special treatment by the Swift compiler? 
 This could also enable more dynamic DSL like features.
>>> 
>>> I haven’t explored enough of the design space to be sure, but I’d want to 
>>> make sure that a library version of this could be done without giving up 
>>> ergonomics of the result.  If you were interested in being able to interop 
>>> with other languages that are dynamically typed and reference counted, then 
>>> something like this could be possible in principle:
>> 
>> Thinking about the Perl case makes it clear to me that this should not be 
>> built into the compiler as a monolithic thing.  Perl supports several 
>> different types (SV/AV/HV) which represent different concepts (scalars, 
>> arrays, hashes) so baking it all together into one thing would be the wrong 
>> way to map it.  In fact, the magic we need is pretty small, and seems 
>> generally useful for other things. Consider a design like this:
>> 
>> 
>> // not magic, things like Int, String and many other conform to this. 
>> protocol Pythonable {
>>  init?(_ : PythonObject)
>>  func toPython() -> PythonObject
>> }
> 
> It’s not magic unless you expect the compiler or runtime to help with 
> conversion between Int/String/etc. and PythonObject, as with 
> _ObjectiveCBridgeable.
> 
>> 
>> // Not magic.
>> struct PythonObject : /*protocols below*/ {
>>   var state : UnsafePointer
>> 
>>   subscript(_ : Pythonable…) -> PythonObject {
>> ...
>>   }
>> }
>> 
>> // Magic, must be on the struct definition.  
>> // Could alternatively allow custom copy/move/… ctors like C++.
>> protocol CustomValueWitnessTable {
>>  static func init(..)
>>  static func copy(..)
>>  static func move(..)
>>  static func destroy(..)
>> }
> 
> Swift’s implementation model supports this. As a surface-level construct it’s 
> going to be mired in UnsafeMutablePointers, and it’s not at all clear to me 
> that we want this level of control there. Presumably, binding to Python is 
> going to require some compiler effort—defining how it is that Python objects 
> are initialized/copied/moved/destroyed seems like a reasonable part of that 
> effort.
> 
>> // Magic, allows anyobject-like member lookup on a type when lookup 
>> otherwise fails.
>> protocol DynamicMemberLookupable {
>>   associatedtype MemberLookupResultType
>>   func dynamicMemberLookup(_ : String) -> MemberLookupResultType
>> }
> 
> AnyObject lookup looks for an actual declaration on any type anywhere. One 
> could extend that mechanism to, say, return all Python methods and assume 
> that you can call any Python method with any PythonObject instance. AnyObject 
> lookup is fairly unprincipled as a language feature, because there’s no 
> natural scope in which to perform name lookup, and involves hacks at many 
> levels that don’t always work (e.g., AnyObject lookup… sometimes… fails 
> across multiple source files for hard-to-explain reasons). You’re taking 

Re: [swift-evolution] [Discussion] Swift for Data Science / ML / Big Data analytics

[Douglas Gregor via swift-evolution]

> On Oct 29, 2017, at 1:34 PM, Chris Lattner via swift-evolution 
>  wrote:
> 
>> 
>> On Oct 29, 2017, at 8:23 AM, Chris Lattner via swift-evolution 
>>  wrote:
>> 
>> 
>>> On Oct 29, 2017, at 4:04 AM, Lukas Stabe  wrote:
>>> 
 On 28. Oct 2017, at 23:10, Chris Lattner via swift-evolution 
  wrote:
 
 … which is to say, exactly identical to the Python version except that new 
 variables need to be declared with let/var.  This can be done by blessing 
 Python.Object (which is identical to “PyObject*” at the machine level) 
 with some special dynamic name lookup behavior:  Dot syntax turns into a 
 call to PyObject_GetAttrString, subscripts turn into PyObject_GetItem, 
 calls turn into PyObject_Call, etc.  ARC would be implemented with INCREF 
 etc.
>>> 
>>> That sounds like a very interesting prospect. Do you think it would make 
>>> sense to make the language features that facilitate this (dynamic dispatch 
>>> of method calls, property accesses, subscript and ARC) available to Swift 
>>> classes annotated in some way, so that interop like this can be implemented 
>>> as a library without special treatment by the Swift compiler? This could 
>>> also enable more dynamic DSL like features.
>> 
>> I haven’t explored enough of the design space to be sure, but I’d want to 
>> make sure that a library version of this could be done without giving up 
>> ergonomics of the result.  If you were interested in being able to interop 
>> with other languages that are dynamically typed and reference counted, then 
>> something like this could be possible in principle:
> 
> Thinking about the Perl case makes it clear to me that this should not be 
> built into the compiler as a monolithic thing.  Perl supports several 
> different types (SV/AV/HV) which represent different concepts (scalars, 
> arrays, hashes) so baking it all together into one thing would be the wrong 
> way to map it.  In fact, the magic we need is pretty small, and seems 
> generally useful for other things. Consider a design like this:
> 
> 
> // not magic, things like Int, String and many other conform to this. 
> protocol Pythonable {
>  init?(_ : PythonObject)
>  func toPython() -> PythonObject
> }

It’s not magic unless you expect the compiler or runtime to help with 
conversion between Int/String/etc. and PythonObject, as with 
_ObjectiveCBridgeable.

> 
> // Not magic.
> struct PythonObject : /*protocols below*/ {
>   var state : UnsafePointer
> 
>   subscript(_ : Pythonable…) -> PythonObject {
> ...
>   }
> }
> 
> // Magic, must be on the struct definition.  
> // Could alternatively allow custom copy/move/… ctors like C++.
> protocol CustomValueWitnessTable {
>  static func init(..)
>  static func copy(..)
>  static func move(..)
>  static func destroy(..)
> }

Swift’s implementation model supports this. As a surface-level construct it’s 
going to be mired in UnsafeMutablePointers, and it’s not at all clear to me 
that we want this level of control there. Presumably, binding to Python is 
going to require some compiler effort—defining how it is that Python objects 
are initialized/copied/moved/destroyed seems like a reasonable part of that 
effort.

> // Magic, allows anyobject-like member lookup on a type when lookup otherwise 
> fails.
> protocol DynamicMemberLookupable {
>   associatedtype MemberLookupResultType
>   func dynamicMemberLookup(_ : String) -> MemberLookupResultType
> }

AnyObject lookup looks for an actual declaration on any type anywhere. One 
could extend that mechanism to, say, return all Python methods and assume that 
you can call any Python method with any PythonObject instance. AnyObject lookup 
is fairly unprincipled as a language feature, because there’s no natural scope 
in which to perform name lookup, and involves hacks at many levels that don’t 
always work (e.g., AnyObject lookup… sometimes… fails across multiple source 
files for hard-to-explain reasons). You’re taking on that brokenness if you 
expand AnyObject lookup to another ecosystem.

Although it doesn’t really seem like AnyObject lookup is the thing you’re 
asking for here. It seems more like you want dynamicMemberLookup(_:) to capture 
“self” and the method name, and then be a callable thing as below...

> 
> // Magic, allows “overloaded/sugared postfix ()”.
> protocol CustomCallable {
>  func call( …)
> }
> 
> The only tricky thing about this is the call part of things.  At least in the 
> case of python, we want something like this:
> 
>   foo.bar(1, 2, a: x, b: y)
> 
> to turn into:
>  foo.dynamicMemberLookup(“bar”).call(1, 2, kwargs: [“a”:x, “b”:y])
> 
> We don’t want this to be a memberlookup of a value that has “bar” as a 
> basename and “a:” and “b:” as parameter labels.

Well, I think the MemberLookupResult is going to get the name “bar”, argument 
labels “_:_:a:b:”, and arguments “1”, “2”, “x”, “y”, because that’s the Swift 
model of argument labels. It can then reshuffle them however it needs to for 

Re: [swift-evolution] [Discussion] Swift for Data Science / ML / Big Data analytics

[Lukas Stabe via swift-evolution]

> On 29. Oct 2017, at 21:34, Chris Lattner  wrote:
> 
> // Magic, allows “overloaded/sugared postfix ()”.
> protocol CustomCallable {
>  func call( …)
> }
> 
> The only tricky thing about this is the call part of things.  At least in the 
> case of python, we want something like this:
> 
>   foo.bar(1, 2, a: x, b: y)
> 
> to turn into:
>  foo.dynamicMemberLookup(“bar”).call(1, 2, kwargs: [“a”:x, “b”:y])
> 
> We don’t want this to be a memberlookup of a value that has “bar” as a 
> basename and “a:” and “b:” as parameter labels.

I don't know much about Swift internals, but since you can already access a 
function member without calling it, the parsing part of this sounds like it 
would be straightforward.

If CustomCallable looked something like this:

protocol CustomCallable {
associatedtype Parameter
associatedtype Result
func call(_ parameterList: [(label: String?, value: Parameter)]) -> Result
}

implementations could decide how they want to handle named/unnamed parameters 
etc themselves.

This approach would currently come with some limitations though: No throwing 
(this could be addressed by a separate CustomThrowingCallable protocol), and – 
since currently non-nominal types can't adopt protocols – an implementation of 
CustomCallable could not accept, for example, both normal values and 
closures/tuples.

>> Are there other uses for such a thing?

DynamicMemberLookupable could enable libraries dealing with key-value data, 
like SwiftyJSON for example, to let users write myJson.someKey.nested instead 
of myJson["someKey"]["nested"]. I'm sure there are more uses like this that are 
not integrations with other languages.

— Lukas
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Discussion] Swift for Data Science / ML / Big Data analytics

[C. Keith Ray via swift-evolution]

There's surprisingly little information about Cython interoperability with 
Cocoa/Mac applications. It's might be linkable if the app can embed Python. 

--
C. Keith Ray

* https://leanpub.com/wepntk <- buy my book?
* http://www.thirdfoundationsw.com/keith_ray_resume_2014_long.pdf
* http://agilesolutionspace.blogspot.com/

> On Oct 29, 2017, at 4:00 PM, Chris Lattner  wrote:
> 
> 
>> On Oct 29, 2017, at 3:23 PM, C. Keith Ray  wrote:
>> 
>> It would be really nice if Cython code could be linked into iOS and Mac apps 
>> written in Swift or Objective-C.
>> 
>> http://cython.org
> 
> Isn’t that possible today, at least on the Mac?
> 
> -Chris
> 
> 
>> 
>> --
>> C. Keith Ray
>> Senior Software Engineer / Trainer / Agile Coach
>> * http://www.thirdfoundationsw.com/keith_ray_resume_2014_long.pdf
>> 
>> 
>> 
 On Oct 29, 2017, at 1:34 PM, Chris Lattner via swift-evolution 
  wrote:
 
 
> On Oct 29, 2017, at 8:23 AM, Chris Lattner via swift-evolution 
>  wrote:
> 
> 
>> On Oct 29, 2017, at 4:04 AM, Lukas Stabe  wrote:
>> 
>> On 28. Oct 2017, at 23:10, Chris Lattner via swift-evolution 
>>  wrote:
>> 
>> … which is to say, exactly identical to the Python version except that 
>> new variables need to be declared with let/var.  This can be done by 
>> blessing Python.Object (which is identical to “PyObject*” at the machine 
>> level) with some special dynamic name lookup behavior:  Dot syntax turns 
>> into a call to PyObject_GetAttrString, subscripts turn into 
>> PyObject_GetItem, calls turn into PyObject_Call, etc. ARC would be 
>> implemented with INCREF etc.
> 
> That sounds like a very interesting prospect. Do you think it would make 
> sense to make the language features that facilitate this (dynamic 
> dispatch of method calls, property accesses, subscript and ARC) available 
> to Swift classes annotated in some way, so that interop like this can be 
> implemented as a library without special treatment by the Swift compiler? 
> This could also enable more dynamic DSL like features.
 
 I haven’t explored enough of the design space to be sure, but I’d want to 
 make sure that a library version of this could be done without giving up 
 ergonomics of the result.  If you were interested in being able to interop 
 with other languages that are dynamically typed and reference counted, 
 then something like this could be possible in principle:
>>> 
>>> Thinking about the Perl case makes it clear to me that this should not be 
>>> built into the compiler as a monolithic thing.  Perl supports several 
>>> different types (SV/AV/HV) which represent different concepts (scalars, 
>>> arrays, hashes) so baking it all together into one thing would be the wrong 
>>> way to map it.  In fact, the magic we need is pretty small, and seems 
>>> generally useful for other things.  Consider a design like this:
>>> 
>>> 
>>> // not magic, things like Int, String and many other conform to this. 
>>> protocol Pythonable {
>>>  init?(_ : PythonObject)
>>>  func toPython() -> PythonObject
>>> }
>>> 
>>> // Not magic.
>>> struct PythonObject : /*protocols below*/ {
>>>   var state : UnsafePointer
>>> 
>>>   subscript(_ : Pythonable…) -> PythonObject {
>>> ...
>>>   }
>>> }
>>> 
>>> // Magic, must be on the struct definition.  
>>> // Could alternatively allow custom copy/move/… ctors like C++.
>>> protocol CustomValueWitnessTable {
>>>  static func init(..)
>>>  static func copy(..)
>>>  static func move(..)
>>>  static func destroy(..)
>>> }
>>> 
>>> // Magic, allows anyobject-like member lookup on a type when lookup 
>>> otherwise fails.
>>> protocol DynamicMemberLookupable {
>>>   associatedtype MemberLookupResultType
>>>   func dynamicMemberLookup(_ : String) -> MemberLookupResultType
>>> }
>>> 
>>> // Magic, allows “overloaded/sugared postfix ()”.
>>> protocol CustomCallable {
>>>  func call( …)
>>> }
>>> 
>>> The only tricky thing about this is the call part of things.  At least in 
>>> the case of python, we want something like this:
>>> 
>>>   foo.bar(1, 2, a: x, b: y)
>>> 
>>> to turn into:
>>>  foo.dynamicMemberLookup(“bar”).call(1, 2, kwargs: [“a”:x, “b”:y])
>>> 
>>> We don’t want this to be a memberlookup of a value that has “bar” as a 
>>> basename and “a:” and “b:” as parameter labels.
>>> 
>>> -Chris
>>> 
 
 protocol DynamicDispatchable { // Protocol is “magic" known by the 
 compiler.
 func retain()
 func release()
 func memberLookup(_ : String) -> Self
 func subscript(_ : T) -> Self
 func call(_ args: [Self]) -> Self
 } 
 
 module Python {
 struct Object : DynamicDispatchable {
   var state : UnsafePointer
 
   func retain() {
  INCREF(self)
  }
 
func memberLookup(_ : String) -> Object {
   PyObject_GetAttrString(…)
}
   etc
 }
 
 module Perl5 { 
  struct Object : Dynam

Re: [swift-evolution] [Discussion] Swift for Data Science / ML / Big Data analytics

[Chris Lattner via swift-evolution]

> On Oct 29, 2017, at 3:23 PM, C. Keith Ray  wrote:
> 
> It would be really nice if Cython code could be linked into iOS and Mac apps 
> written in Swift or Objective-C.
> 
> http://cython.org 
Isn’t that possible today, at least on the Mac?

-Chris


> 
> --
> C. Keith Ray
> Senior Software Engineer / Trainer / Agile Coach
> * http://www.thirdfoundationsw.com/keith_ray_resume_2014_long.pdf 
> 
> 
> 
> 
>> On Oct 29, 2017, at 1:34 PM, Chris Lattner via swift-evolution 
>> mailto:swift-evolution@swift.org>> wrote:
>> 
>>> 
>>> On Oct 29, 2017, at 8:23 AM, Chris Lattner via swift-evolution 
>>> mailto:swift-evolution@swift.org>> wrote:
>>> 
>>> 
 On Oct 29, 2017, at 4:04 AM, Lukas Stabe >>> > wrote:
 
> On 28. Oct 2017, at 23:10, Chris Lattner via swift-evolution 
> mailto:swift-evolution@swift.org>> wrote:
> 
> … which is to say, exactly identical to the Python version except that 
> new variables need to be declared with let/var.  This can be done by 
> blessing Python.Object (which is identical to “PyObject*” at the machine 
> level) with some special dynamic name lookup behavior:  Dot syntax turns 
> into a call to PyObject_GetAttrString, subscripts turn into 
> PyObject_GetItem, calls turn into PyObject_Call, etc. ARC would be 
> implemented with INCREF etc.
 
 That sounds like a very interesting prospect. Do you think it would make 
 sense to make the language features that facilitate this (dynamic dispatch 
 of method calls, property accesses, subscript and ARC) available to Swift 
 classes annotated in some way, so that interop like this can be 
 implemented as a library without special treatment by the Swift compiler? 
 This could also enable more dynamic DSL like features.
>>> 
>>> I haven’t explored enough of the design space to be sure, but I’d want to 
>>> make sure that a library version of this could be done without giving up 
>>> ergonomics of the result.  If you were interested in being able to interop 
>>> with other languages that are dynamically typed and reference counted, then 
>>> something like this could be possible in principle:
>> 
>> Thinking about the Perl case makes it clear to me that this should not be 
>> built into the compiler as a monolithic thing.  Perl supports several 
>> different types (SV/AV/HV) which represent different concepts (scalars, 
>> arrays, hashes) so baking it all together into one thing would be the wrong 
>> way to map it.  In fact, the magic we need is pretty small, and seems 
>> generally useful for other things.  Consider a design like this:
>> 
>> 
>> // not magic, things like Int, String and many other conform to this. 
>> protocol Pythonable {
>>  init?(_ : PythonObject)
>>  func toPython() -> PythonObject
>> }
>> 
>> // Not magic.
>> struct PythonObject : /*protocols below*/ {
>>   var state : UnsafePointer
>> 
>>   subscript(_ : Pythonable…) -> PythonObject {
>> ...
>>   }
>> }
>> 
>> // Magic, must be on the struct definition.  
>> // Could alternatively allow custom copy/move/… ctors like C++.
>> protocol CustomValueWitnessTable {
>>  static func init(..)
>>  static func copy(..)
>>  static func move(..)
>>  static func destroy(..)
>> }
>> 
>> // Magic, allows anyobject-like member lookup on a type when lookup 
>> otherwise fails.
>> protocol DynamicMemberLookupable {
>>   associatedtype MemberLookupResultType
>>   func dynamicMemberLookup(_ : String) -> MemberLookupResultType
>> }
>> 
>> // Magic, allows “overloaded/sugared postfix ()”.
>> protocol CustomCallable {
>>  func call( …)
>> }
>> 
>> The only tricky thing about this is the call part of things.  At least in 
>> the case of python, we want something like this:
>> 
>>   foo.bar(1, 2, a: x, b: y)
>> 
>> to turn into:
>>  foo.dynamicMemberLookup(“bar”).call(1, 2, kwargs: [“a”:x, “b”:y])
>> 
>> We don’t want this to be a memberlookup of a value that has “bar” as a 
>> basename and “a:” and “b:” as parameter labels.
>> 
>> -Chris
>> 
>>> 
>>> protocol DynamicDispatchable { // Protocol is “magic" known by the compiler.
>>> func retain()
>>> func release()
>>> func memberLookup(_ : String) -> Self
>>> func subscript(_ : T) -> Self
>>> func call(_ args: [Self]) -> Self
>>> } 
>>> 
>>> module Python {
>>> struct Object : DynamicDispatchable {
>>>   var state : UnsafePointer
>>> 
>>>   func retain() {
>>>  INCREF(self)
>>>  }
>>> 
>>>func memberLookup(_ : String) -> Object {
>>>   PyObject_GetAttrString(…)
>>>}
>>>   etc
>>> }
>>> 
>>> module Perl5 { 
>>>  struct Object : DynamicDispatchable {
>>>   var state : UnsafePointer
>>> 
>>>   func retain() {
>>>  SvREFCNT_inc(self)
>>>  }
>>> ….
>>> 
>>> 
>>> 
>>> Are there other uses for such a thing?
>>> 
>>> -Chris
>>> 
>>> 
>>> ___
>>> swift-evolution mailing list
>>> swift-evolution@swift.org 

Re: [swift-evolution] [Discussion] Swift for Data Science / ML / Big Data analytics

[C. Keith Ray via swift-evolution]

It would be really nice if Cython code could be linked into iOS and Mac apps 
written in Swift or Objective-C.

http://cython.org

--
C. Keith Ray
Senior Software Engineer / Trainer / Agile Coach
* http://www.thirdfoundationsw.com/keith_ray_resume_2014_long.pdf



> On Oct 29, 2017, at 1:34 PM, Chris Lattner via swift-evolution 
>  wrote:
> 
>> 
>> On Oct 29, 2017, at 8:23 AM, Chris Lattner via swift-evolution 
>>  wrote:
>> 
>> 
>>> On Oct 29, 2017, at 4:04 AM, Lukas Stabe  wrote:
>>> 
 On 28. Oct 2017, at 23:10, Chris Lattner via swift-evolution 
  wrote:
 
 … which is to say, exactly identical to the Python version except that new 
 variables need to be declared with let/var.  This can be done by blessing 
 Python.Object (which is identical to “PyObject*” at the machine level) 
 with some special dynamic name lookup behavior:  Dot syntax turns into a 
 call to PyObject_GetAttrString, subscripts turn into PyObject_GetItem, 
 calls turn into PyObject_Call, etc. ARC would be implemented with INCREF 
 etc.
>>> 
>>> That sounds like a very interesting prospect. Do you think it would make 
>>> sense to make the language features that facilitate this (dynamic dispatch 
>>> of method calls, property accesses, subscript and ARC) available to Swift 
>>> classes annotated in some way, so that interop like this can be implemented 
>>> as a library without special treatment by the Swift compiler? This could 
>>> also enable more dynamic DSL like features.
>> 
>> I haven’t explored enough of the design space to be sure, but I’d want to 
>> make sure that a library version of this could be done without giving up 
>> ergonomics of the result.  If you were interested in being able to interop 
>> with other languages that are dynamically typed and reference counted, then 
>> something like this could be possible in principle:
> 
> Thinking about the Perl case makes it clear to me that this should not be 
> built into the compiler as a monolithic thing.  Perl supports several 
> different types (SV/AV/HV) which represent different concepts (scalars, 
> arrays, hashes) so baking it all together into one thing would be the wrong 
> way to map it.  In fact, the magic we need is pretty small, and seems 
> generally useful for other things.  Consider a design like this:
> 
> 
> // not magic, things like Int, String and many other conform to this. 
> protocol Pythonable {
>  init?(_ : PythonObject)
>  func toPython() -> PythonObject
> }
> 
> // Not magic.
> struct PythonObject : /*protocols below*/ {
>   var state : UnsafePointer
> 
>   subscript(_ : Pythonable…) -> PythonObject {
> ...
>   }
> }
> 
> // Magic, must be on the struct definition.  
> // Could alternatively allow custom copy/move/… ctors like C++.
> protocol CustomValueWitnessTable {
>  static func init(..)
>  static func copy(..)
>  static func move(..)
>  static func destroy(..)
> }
> 
> // Magic, allows anyobject-like member lookup on a type when lookup otherwise 
> fails.
> protocol DynamicMemberLookupable {
>   associatedtype MemberLookupResultType
>   func dynamicMemberLookup(_ : String) -> MemberLookupResultType
> }
> 
> // Magic, allows “overloaded/sugared postfix ()”.
> protocol CustomCallable {
>  func call( …)
> }
> 
> The only tricky thing about this is the call part of things.  At least in the 
> case of python, we want something like this:
> 
>   foo.bar(1, 2, a: x, b: y)
> 
> to turn into:
>  foo.dynamicMemberLookup(“bar”).call(1, 2, kwargs: [“a”:x, “b”:y])
> 
> We don’t want this to be a memberlookup of a value that has “bar” as a 
> basename and “a:” and “b:” as parameter labels.
> 
> -Chris
> 
>> 
>> protocol DynamicDispatchable { // Protocol is “magic" known by the compiler.
>> func retain()
>> func release()
>> func memberLookup(_ : String) -> Self
>> func subscript(_ : T) -> Self
>> func call(_ args: [Self]) -> Self
>> } 
>> 
>> module Python {
>> struct Object : DynamicDispatchable {
>>   var state : UnsafePointer
>> 
>>   func retain() {
>>  INCREF(self)
>>  }
>> 
>>func memberLookup(_ : String) -> Object {
>>   PyObject_GetAttrString(…)
>>}
>>   etc
>> }
>> 
>> module Perl5 { 
>>  struct Object : DynamicDispatchable {
>>   var state : UnsafePointer
>> 
>>   func retain() {
>>  SvREFCNT_inc(self)
>>  }
>> ….
>> 
>> 
>> 
>> Are there other uses for such a thing?
>> 
>> -Chris
>> 
>> 
>> ___
>> swift-evolution mailing list
>> swift-evolution@swift.org
>> https://lists.swift.org/mailman/listinfo/swift-evolution
> 
> ___
> swift-evolution mailing list
> swift-evolution@swift.org 
> https://lists.swift.org/mailman/listinfo/swift-evolution 
> 
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Discussion] Swift for Data Science / ML / Big Data analytics

[Chris Lattner via swift-evolution]

> On Oct 29, 2017, at 8:23 AM, Chris Lattner via swift-evolution 
>  wrote:
> 
> 
>> On Oct 29, 2017, at 4:04 AM, Lukas Stabe  wrote:
>> 
>>> On 28. Oct 2017, at 23:10, Chris Lattner via swift-evolution 
>>>  wrote:
>>> 
>>> … which is to say, exactly identical to the Python version except that new 
>>> variables need to be declared with let/var.  This can be done by blessing 
>>> Python.Object (which is identical to “PyObject*” at the machine level) with 
>>> some special dynamic name lookup behavior:  Dot syntax turns into a call to 
>>> PyObject_GetAttrString, subscripts turn into PyObject_GetItem, calls turn 
>>> into PyObject_Call, etc.  ARC would be implemented with INCREF etc.
>> 
>> That sounds like a very interesting prospect. Do you think it would make 
>> sense to make the language features that facilitate this (dynamic dispatch 
>> of method calls, property accesses, subscript and ARC) available to Swift 
>> classes annotated in some way, so that interop like this can be implemented 
>> as a library without special treatment by the Swift compiler? This could 
>> also enable more dynamic DSL like features.
> 
> I haven’t explored enough of the design space to be sure, but I’d want to 
> make sure that a library version of this could be done without giving up 
> ergonomics of the result.  If you were interested in being able to interop 
> with other languages that are dynamically typed and reference counted, then 
> something like this could be possible in principle:

Thinking about the Perl case makes it clear to me that this should not be built 
into the compiler as a monolithic thing.  Perl supports several different types 
(SV/AV/HV) which represent different concepts (scalars, arrays, hashes) so 
baking it all together into one thing would be the wrong way to map it.  In 
fact, the magic we need is pretty small, and seems generally useful for other 
things.  Consider a design like this:


// not magic, things like Int, String and many other conform to this. 
protocol Pythonable {
  init?(_ : PythonObject)
  func toPython() -> PythonObject
}

// Not magic.
struct PythonObject : /*protocols below*/ {
   var state : UnsafePointer

   subscript(_ : Pythonable…) -> PythonObject {
 ...
   }
}

// Magic, must be on the struct definition.  
// Could alternatively allow custom copy/move/… ctors like C++.
protocol CustomValueWitnessTable {
  static func init(..)
  static func copy(..)
  static func move(..)
  static func destroy(..)
}

// Magic, allows anyobject-like member lookup on a type when lookup otherwise 
fails.
protocol DynamicMemberLookupable {
   associatedtype MemberLookupResultType
   func dynamicMemberLookup(_ : String) -> MemberLookupResultType
}

// Magic, allows “overloaded/sugared postfix ()”.
protocol CustomCallable {
  func call( …)
}

The only tricky thing about this is the call part of things.  At least in the 
case of python, we want something like this:

   foo.bar(1, 2, a: x, b: y)

to turn into:
  foo.dynamicMemberLookup(“bar”).call(1, 2, kwargs: [“a”:x, “b”:y])

We don’t want this to be a memberlookup of a value that has “bar” as a basename 
and “a:” and “b:” as parameter labels.

-Chris

> 
> protocol DynamicDispatchable { // Protocol is “magic" known by the compiler.
>  func retain()
>  func release()
>  func memberLookup(_ : String) -> Self
>  func subscript(_ : T) -> Self
>  func call(_ args: [Self]) -> Self
> } 
> 
> module Python {
>  struct Object : DynamicDispatchable {
>var state : UnsafePointer
> 
>func retain() {
>   INCREF(self)
>   }
> 
> func memberLookup(_ : String) -> Object {
>PyObject_GetAttrString(…)
> }
>etc
>  }
> 
> module Perl5 { 
>   struct Object : DynamicDispatchable {
>var state : UnsafePointer
> 
>func retain() {
>   SvREFCNT_inc(self)
>   }
> ….
> 
> 
> 
> Are there other uses for such a thing?
> 
> -Chris
> 
> 
> ___
> swift-evolution mailing list
> swift-evolution@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Discussion] Swift for Data Science / ML / Big Data analytics

[Chris Lattner via swift-evolution]

> On Oct 29, 2017, at 4:04 AM, Lukas Stabe  wrote:
> 
>> On 28. Oct 2017, at 23:10, Chris Lattner via swift-evolution 
>>  wrote:
>> 
>> … which is to say, exactly identical to the Python version except that new 
>> variables need to be declared with let/var.  This can be done by blessing 
>> Python.Object (which is identical to “PyObject*” at the machine level) with 
>> some special dynamic name lookup behavior:  Dot syntax turns into a call to 
>> PyObject_GetAttrString, subscripts turn into PyObject_GetItem, calls turn 
>> into PyObject_Call, etc.  ARC would be implemented with INCREF etc.
> 
> That sounds like a very interesting prospect. Do you think it would make 
> sense to make the language features that facilitate this (dynamic dispatch of 
> method calls, property accesses, subscript and ARC) available to Swift 
> classes annotated in some way, so that interop like this can be implemented 
> as a library without special treatment by the Swift compiler? This could also 
> enable more dynamic DSL like features.

I haven’t explored enough of the design space to be sure, but I’d want to make 
sure that a library version of this could be done without giving up ergonomics 
of the result.  If you were interested in being able to interop with other 
languages that are dynamically typed and reference counted, then something like 
this could be possible in principle:

protocol DynamicDispatchable { // Protocol is “magic" known by the compiler.
  func retain()
  func release()
  func memberLookup(_ : String) -> Self
  func subscript(_ : T) -> Self
  func call(_ args: [Self]) -> Self
} 

module Python {
  struct Object : DynamicDispatchable {
var state : UnsafePointer

func retain() {
   INCREF(self)
   }

 func memberLookup(_ : String) -> Object {
PyObject_GetAttrString(…)
 }
etc
  }

module Perl5 { 
   struct Object : DynamicDispatchable {
var state : UnsafePointer

func retain() {
   SvREFCNT_inc(self)
   }
….



Are there other uses for such a thing?

-Chris


___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Discussion] Swift for Data Science / ML / Big Data analytics

[Lukas Stabe via swift-evolution]

> On 28. Oct 2017, at 23:10, Chris Lattner via swift-evolution 
>  wrote:
> 
> … which is to say, exactly identical to the Python version except that new 
> variables need to be declared with let/var.  This can be done by blessing 
> Python.Object (which is identical to “PyObject*” at the machine level) with 
> some special dynamic name lookup behavior:  Dot syntax turns into a call to 
> PyObject_GetAttrString, subscripts turn into PyObject_GetItem, calls turn 
> into PyObject_Call, etc.  ARC would be implemented with INCREF etc.

That sounds like a very interesting prospect. Do you think it would make sense 
to make the language features that facilitate this (dynamic dispatch of method 
calls, property accesses, subscript and ARC) available to Swift classes 
annotated in some way, so that interop like this can be implemented as a 
library without special treatment by the Swift compiler? This could also enable 
more dynamic DSL like features.

— Lukas


signature.asc
Description: Message signed with OpenPGP
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

Re: [swift-evolution] [Discussion] Swift for Data Science / ML / Big Data analytics

[Chris Lattner via swift-evolution]

On Oct 28, 2017, at 9:45 AM, Maxim Veksler via swift-evolution 
 wrote:
> 
> Hey Guys,
> 
> The big data and machine learning world is dominated by Python, Scala an R. 
> 
> I'm a Swifter by heart, but not so much by tools of trait. 

Hi Max,

I’m very interested in this topic, with a specific focus on Python.  It isn’t 
the immediate thing on my priority list to deal with, but I hope that we get to 
push on this.

In short, I think we should build a simple Swift/Python interop story.  This 
sort of thing has be built numerous times for many languages (owing to Python’s 
great support for embed-ability), including things like PyObjC, boost.python, 
and many others.

In Swift, it is straightforward to make this example 
(http://cs231n.github.io/python-numpy-tutorial/#numpy-arrays 
) look something 
like this:

let np = Python.import(“numpy”)   // Returns a value of type 
Python.Object.
let a = np.array([1, 2, 3])
print(type(a))// Whether we want to support type(x) or use the 
Swift equivalent would be up for discussion of course!
print(a.shape)
print(a[0], a[1], a[2])
a[0] = 5
print(a)

let b = np.array([[1,2,3],[4,5,6]])
print(b.shape)
print(b[0, 0], b[0, 1], b[1, 0])

… which is to say, exactly identical to the Python version except that new 
variables need to be declared with let/var.  This can be done by blessing 
Python.Object (which is identical to “PyObject*” at the machine level) with 
some special dynamic name lookup behavior:  Dot syntax turns into a call to 
PyObject_GetAttrString, subscripts turn into PyObject_GetItem, calls turn into 
PyObject_Call, etc.  ARC would be implemented with INCREF etc.

If we do this, the vast majority of the Python ecosystem should be directly 
usable from within Swift code, and the only a few major syntactic differences 
(e.g. ranges work differently).  We would add failable inits to the primitive 
datatypes like Int/String/etc to convert Python.Object values into them, and 
add the corresponding non-failable conversions from Python.Object to those 
primitives.

Overall, I think it will provide a really nice experience, and allow us to 
leverage the vast majority of the Python ecosystem directly in Swift code. This 
project would also have much more narrow impact on the Swift compiler than the 
ObjC importer (since it works completely differently).  For a first cut, I 
don’t think we would have to worry about Swift classes subclassing Python 
classes, for example.

-Chris





> 
> I'd appreciate a constructive discussion on how that could be changed.
> 
> While R is a non goal for obvious reasons, i'd argue that since both Scala 
> and Python are general purpose languages, taking them head to head might be a 
> low hanging fruit.
> 
> To make the claim I'd like to reference to projects such as 
> 
>  - Hadoop, Spark, Hive are all huge eco-systems which are entirely JVM based.
>  - Apache Parquet, a highly efficient column based storage format for big 
> data analytics which was implemented in Java, and C++.
>  - Apache Arrow, a physical memory spec that big data systems can use to 
> allow zero transformations on data transferred between systems. Which (for 
> obvious reasons) focused on JVM, to C interoperability.
> 
> Python's Buffer Protocol which ensures it's predominance (for the time being) 
> as a prime candidate for data science related projects 
> https://jeffknupp.com/blog/2017/09/15/python-is-the-fastest-growing-programming-language-due-to-a-feature-youve-never-heard-of/
>  
> 
> 
> While Swift's Memory Ownership manifesto touches similar turf discussing copy 
> on write and optimizing memory access overhead it IMHO takes a system level 
> perspective targeting projects such as kernel code. I'd suggest that viewing 
> the problem from an efficient CPU/GPU data crunching machine perspective 
> might shade a different light on the requirements and use cases. 
> 
> 
> I'd be happy to learn more, and have a constructive discussion on the subject.
> 
> 
> Thank you,
> Max.
>  
> 
> -- 
> puıɯ ʎɯ ɯoɹɟ ʇuǝs
> ___
> swift-evolution mailing list
> swift-evolution@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution

[swift-evolution] [Discussion] Swift for Data Science / ML / Big Data analytics

[Maxim Veksler via swift-evolution]

Hey Guys,

The big data and machine learning world is dominated by Python, Scala an R.

I'm a Swifter by heart, but not so much by tools of trait.

I'd appreciate a constructive discussion on how that could be changed.

While R is a non goal for obvious reasons, i'd argue that since both Scala
and Python are general purpose languages, taking them head to head might be
a low hanging fruit.

To make the claim I'd like to reference to projects such as

 - Hadoop, Spark, Hive are all huge eco-systems which are entirely JVM
based.
 - Apache Parquet, a highly efficient column based storage format for big
data analytics which was implemented in Java, and C++.
 - Apache Arrow, a physical memory spec that big data systems can use to
allow zero transformations on data transferred between systems. Which (for
obvious reasons) focused on JVM, to C interoperability.

Python's Buffer Protocol which ensures it's predominance (for the time
being) as a prime candidate for data science related projects
https://jeffknupp.com/blog/2017/09/15/python-is-the-
fastest-growing-programming-language-due-to-a-feature-youve-never-heard-of/

While Swift's Memory Ownership manifesto touches similar turf discussing
copy on write and optimizing memory access overhead it IMHO takes a system
level perspective targeting projects such as kernel code. I'd suggest that
viewing the problem from an efficient CPU/GPU data crunching machine
perspective might shade a different light on the requirements and use
cases.


I'd be happy to learn more, and have a constructive discussion on the
subject.


Thank you,
Max.


-- 
puıɯ ʎɯ ɯoɹɟ ʇuǝs
___
swift-evolution mailing list
swift-evolution@swift.org
https://lists.swift.org/mailman/listinfo/swift-evolution