Ok, made some progress. Still having trouble with a _ENV_ not defined
error: how could it not be defined if it's a global variable??? Figuring
out argument passing is going to be tricky. Is this kind of system feasible
or is it going to cause huge slowdowns?
using DataFrames
import Base.convert
_TYPES_ = Symbol[]
type _LAZY_
_E_::Expr
_ENV_::Symbol
end
# modify dicts such that if a key is not found, search the parent
function Base.getindex{K,V}(h::Dict{K,V}, key)
index = Base.ht_keyindex(h, key)
if index < 0
if :_parent in keys(h)
Base.getindex(eval(h[:_parent]), key)
else
throw(KeyError(key))
end
else
h.vals[index]::V
end
end
# allow inheritance from modules
function convert(::Type{Dict}, m::Module)
dict = Dict()
for name in names(m)
dict[name] = eval( :(Base.$name) )
# add types to type list
if typeof(dict[name]) <: DataType
push!(_TYPES_, name)
end
end
dict
end
# allow inheritance from DataFrames
function convert(::Type{Dict}, d::DataFrame)
dict = Dict()
for name in names(d)
dict[name] = d[name]
# add types to type list
if typeof(dict[name]) <: DataType
push!(_TYPES_, name)
end
end
dict
end
# establish the base environment, save it as global, and point it to an
empty dictionary
macro _ENV_MACRO_()
esc(quote
_ENV_ = gensym()
_GLOBAL_ = _ENV_
eval(quote
$_ENV_ = Dict()
end)
end)
end
# establish a new environment and point it towards a dict daughter of the
old environment
macro _NEW_ENV_MACRO_()
esc(quote
_NEW_ENV_ = gensym()
eval(quote
$_NEW_ENV_ = {:_parent => $(Expr(:quote, _ENV_)) }
end)
_ENV_ = _NEW_ENV_
end)
end
# establish a new environment and point it towards a dict daughter of the
old environment with dict contents
macro _ADD_ENV_MACRO_(dict)
esc(quote
_DICT_ = $dict
_ADD_ENV_ = gensym()
eval(quote
$_ADD_ENV_ = convert(Dict, $_DICT_)
$_ADD_ENV_[:_parent] = $(Expr(:quote, _ENV_))
end)
_ENV_ = _ADD_ENV_
end)
end
# jump back in time to the previous generation
macro _REMOVE_ENV_MACRO_()
esc(quote
_ENV_ = eval(_ENV_)[:_parent]
end)
end
# new types will have to be included in a module at the beginning of code
# that module will need to be converted to a Dict along with base
# a namespace will need to be created such that module dicts inherit from
each other, with base at the top
@_ENV_MACRO_()
@_ADD_ENV_MACRO_(Base)
@_NEW_ENV_MACRO_
# test expression
e =
quote
a = 1
b = 2
test = function()
b = a
end
end
# reformat code to use dict scoping
function _ENV_REPLACE_(_Lazy_::_LAZY_)
e = copy(_Lazy_._E_)
_ENV_ = _Lazy_._ENV_
# expressions wrapped in _esc will be left alone
if length(e.args) > 0
if (e.head == :call) & (e.args[1] == :_esc)
return e.args[2]
end
end
# set a new scope for a new function. This will also have to be done with
for loops, modules, etc.
if (e.head == :function)
# insert a new scope definition into the function definition
e.args[2].args = [
e.args[2].args[1],
:(@_NEW_ENV_MACRO_),
:(_ENV_REPLACE!_(
$(Expr(:block,
e.args[2].args[2:end]...)))),
:(@_REMOVE_ENV_MACRO_)]
# ignore line numbers
elseif e.head != :line
# for each sentence
for i in 1:length(e.args)
# replace symbols with their dict scoped version
if typeof(e.args[i]) == Symbol
#avoid types
if !(e.args[i] in _TYPES_)
e.args[i] = :($_ENV_[$(string(e.args[i]))])
end
# recur into new expressions
elseif typeof(e.args[i]) == Expr
e.args[i] = _ENV_REPLACE!_(e.args[i], _ENV_)
end
end
end
e
end
function _LAZY_(e::Expr)
_LAZY_(e, _ENV_)
end
macro _LAZY_EVAL_(_Lazy_)
esc(quote
eval(_ENV_REPLACE_($_Lazy_))
end)
end
@_LAZY_EVAL_(_LAZY_(e))
eval(_ENV_)["a"]
eval(_ENV_)["b"]
eval(_ENV_)["test"]() ## ERROR HERE
On Wednesday, July 22, 2015 at 11:30:10 AM UTC+8, Brandon Taylor wrote:
>
> More to do:
> Expressions would also have to be escaped from quoting.
> If we can't scope types within dicts, it might be necessary to have
> special markers for types so they can avoid being scoped.
> I don't think that macros will be necessary anymore
>
>
> On Wednesday, July 22, 2015 at 11:14:38 AM UTC+8, Brandon Taylor wrote:
>>
>> Ok so I've got a good start. I bet John Myles White didn't think I could
>> get this far. Anyway, I'm getting caught up in defining my own escape
>> function. I'm getting lost in multiple layers of meta.
>>
>> using DataFrames
>>
>> import Base.convert
>>
>> # allow inheritance from modules
>> function convert(::Type{Dict}, m::Module)
>> dict = Dict()
>> for name in names(m)
>> dict[name] = eval( :(Base.$name) )
>> end
>> dict
>> end
>>
>> base_dict = convert(Dict, Base)
>>
>> # allow inheritance from DataFrames
>> function convert(::Type{Dict}, d::DataFrame)
>> dict = Dict()
>> for name in names(d)
>> dict[name] = d[name]
>> end
>> dict
>> end
>>
>> # modify dicts such that if a key is not found, search the parent
>> function Base.getindex{K,V}(h::Dict{K,V}, key)
>> index = Base.ht_keyindex(h, key)
>> if index < 0
>> if :_parent in keys(h)
>> Base.getindex(h[:_parent], key)
>> else
>> throw(KeyError(key))
>> end
>> else
>> h.vals[index]::V
>> end
>> end
>>
>> # test expression
>> e =
>> quote
>> a = 1
>> # anonymous functions required for proper scoping
>> test = function()
>> b = a
>> end
>> end
>>
>> # set up the global environment
>> _env = gensym()
>> eval(:($_env = [:_parent => base_dict] ) )
>>
>>
>> #_new_env = _env
>> # this is the code that needs to be escaped
>> #eval(:($_new_env = [:_parent => eval(_env)] ) )
>>
>> # reformat code to use dict scoping
>> function env_replace!(e::Expr,
>> _env::Symbol = _env)
>>
>> # expressions wrapped in _esc will be left alone
>> if length(e.args) > 0
>> if (e.head == :call) & (e.args[1] == :_esc)
>> return e.args[2]
>> end
>> end
>>
>> # set a new scope for a new function. This will also have to be done
>> with for loops, modules, etc.
>> if (e.head == :function)
>> # insert a new scope definition into the function definition
>> _new_env = gensym()
>> e.args[2].args = [
>> e.args[2].args[1],
>> :_esc(), #### need help here ###
>> e.args[2].args[2:end] ]
>> _env = _new_env
>> end
>>
>> # ignore line numbers
>> if e.head != :line
>>
>> for i in 1:length(e.args)
>> # replace symbols with their dict scoped version
>> if typeof(e.args[i]) == Symbol
>> e.args[i] = :($_env[$(string(e.args[i]))])
>>
>> # recur into new expressions
>> elseif typeof(e.args[i]) == Expr
>> e.args[i] = env_replace!(e.args[i], _env)
>> end
>> end
>> end
>> e
>> end
>>
>> # here is an eval that allows evaluation within a certain dict scope
>> function lazy_eval(e::Expr,
>> _env::Symbol = _env)
>> eval(env_replace!(e), _env)
>> end
>>
>>
>> ## TO DO
>> # fix _esc problem
>> # prevent environment symbols from being scoped (perhaps with a special
>> marker)
>> # rescope for, while, try, catch, finally, let, and type
>> # perhaps use fast anonymous to avoid performance slowdowns?
>>
>>
>> On Tuesday, July 21, 2015 at 10:10:58 AM UTC+8, Brandon Taylor wrote:
>>>
>>> And there would need to be a special marker for them, such that if I'm
>>> in function f, f[:a] won't get preprocessed as f[:f][:a]
>>>
>>> On Tuesday, July 21, 2015 at 10:03:08 AM UTC+8, Brandon Taylor wrote:
>>>>
>>>> Although that would probably require nested dicts. Each would have a
>>>> parent dict, and if a lookup isn't found in the current dict, the parent
>>>> dict would be searched.
>>>>
>>>> On Tuesday, July 21, 2015 at 9:53:50 AM UTC+8, Brandon Taylor wrote:
>>>>>
>>>>> I should be possible to preprocess code such that everything is put
>>>>> into a dict based on the name of enclosing function (and global variables
>>>>> will just go into a dict called global).
>>>>>
>>>>> On Tuesday, July 21, 2015 at 9:42:00 AM UTC+8, Brandon Taylor wrote:
>>>>>>
>>>>>> Dicts seem to work pretty well for this kind of thing.
>>>>>>
>>>>>> On Tuesday, July 21, 2015 at 9:38:36 AM UTC+8, Brandon Taylor wrote:
>>>>>>>
>>>>>>> I'm getting a cannot assign variables in other modules error.
>>>>>>>
>>>>>>> On Tuesday, July 21, 2015 at 6:39:44 AM UTC+8, Yichao Yu wrote:
>>>>>>>>
>>>>>>>> On Mon, Jul 20, 2015 at 6:35 PM, Brandon Taylor
>>>>>>>> <brandon....@gmail.com> wrote:
>>>>>>>> > Ok, a thought, Julia has an inbuilt idea of a module. Would it be
>>>>>>>> possible
>>>>>>>> > to hijack this functionality to provide pseudo-environments? That
>>>>>>>> is, never
>>>>>>>> > referring to anything that is not already in an explicit module?
>>>>>>>> And also,
>>>>>>>> > have a data-frame simply be a module?
>>>>>>>>
>>>>>>>> I think this would in principle works. A module is basically what
>>>>>>>> global scope means so all the performance concern applies.
>>>>>>>>
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > On Friday, July 10, 2015 at 11:31:36 PM UTC+8, Brandon Taylor
>>>>>>>> wrote:
>>>>>>>> >>
>>>>>>>> >> I don't know if you came across the vignette?
>>>>>>>> >>
>>>>>>>> http://cran.r-project.org/web/packages/lazyeval/vignettes/lazyeval.html
>>>>>>>>
>>>>>>>> ?
>>>>>>>> >> dplyr uses lazyeval extensively, see
>>>>>>>> >> http://cran.r-project.org/web/packages/dplyr/vignettes/nse.html
>>>>>>>> . The cool
>>>>>>>> >> thing about being able to incorporate this kind of thing in
>>>>>>>> Julia would be
>>>>>>>> >> being able to use the self-reflection capabilities.
>>>>>>>> >>
>>>>>>>> >> On Friday, July 10, 2015 at 10:57:16 AM UTC-4, Cedric St-Jean
>>>>>>>> wrote:
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> >>> On Thursday, July 9, 2015 at 10:35:30 PM UTC-4, Brandon Taylor
>>>>>>>> wrote:
>>>>>>>> >>>>
>>>>>>>> >>>> To walk back in time, you could say something like: compile
>>>>>>>> this like
>>>>>>>> >>>> this was is in line 8. Or compile this like this was in line
>>>>>>>> 5. It seems
>>>>>>>> >>>> like Julia already has some of this functionality in macros.
>>>>>>>> Internal
>>>>>>>> >>>> variables are compiled as if they were in local scope. But
>>>>>>>> escaped
>>>>>>>> >>>> expressions are compiled as if they were in global scope.
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> >>> Could you provide context or a real-world use? I've looked at
>>>>>>>> the
>>>>>>>> >>> lazyeval package, and I'm not entirely sure what it does. Does
>>>>>>>> it provide
>>>>>>>> >>> lazy evaluation for R? That's easy to achieve in Julia (well,
>>>>>>>> sorta).
>>>>>>>> >>> Instead of
>>>>>>>> >>>
>>>>>>>> >>> d = determinant(matrix)
>>>>>>>> >>> ....
>>>>>>>> >>> u = 2 * d
>>>>>>>> >>>
>>>>>>>> >>> you can write
>>>>>>>> >>>
>>>>>>>> >>> d = ()->determinant(matrix)
>>>>>>>> >>> ....
>>>>>>>> >>> u = 2 * d() # determinant is evaluated on use, in the context
>>>>>>>> where it
>>>>>>>> >>> was originally defined
>>>>>>>> >>>
>>>>>>>> >>> With macros this can turn into
>>>>>>>> >>>
>>>>>>>> >>> d = lazy(determinant(matrix))
>>>>>>>> >>>
>>>>>>>> >>> which looks nicer (and also can avoid computing the determinant
>>>>>>>> twice if
>>>>>>>> >>> d() is called twice).
>>>>>>>> >>>
>>>>>>>> >>> Cédric
>>>>>>>> >>>
>>>>>>>> >>>>
>>>>>>>> >>>>
>>>>>>>> >>>> On Thursday, July 9, 2015 at 9:11:05 PM UTC-4, Cedric St-Jean
>>>>>>>> wrote:
>>>>>>>> >>>>>
>>>>>>>> >>>>>
>>>>>>>> >>>>>
>>>>>>>> >>>>> On Thursday, July 9, 2015 at 4:14:32 PM UTC-4, Brandon Taylor
>>>>>>>> wrote:
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> Ok, here's where I'm getting hung up. You said that the
>>>>>>>> compiler
>>>>>>>> >>>>>> figures out the creation/lifetime of all variables at
>>>>>>>> compile time. So does
>>>>>>>> >>>>>> that mean there's a list like:
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> a maps to location 0 and exists from line 3 to line 9
>>>>>>>> >>>>>> b maps to location 1 and exists from line 7 to line 9
>>>>>>>> >>>>>> a maps to location 10 and exists from line 7 to 9?
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> and that to map variables to locations on any particular
>>>>>>>> line, the
>>>>>>>> >>>>>> compiler works its way up the list,
>>>>>>>> >>>>>
>>>>>>>> >>>>>
>>>>>>>> >>>>> Yes, more or less.
>>>>>>>> >>>>>
>>>>>>>> >>>>>>
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> This is perhaps even more helpful than the environment. The
>>>>>>>> >>>>>> environment is immediately and completely determinable at
>>>>>>>> any point in the
>>>>>>>> >>>>>> program. This could make it possible to walk back in time
>>>>>>>> even within the
>>>>>>>> >>>>>> same scope.
>>>>>>>> >>>>>
>>>>>>>> >>>>>
>>>>>>>> >>>>> Could you expand on what you're thinking of?
>>>>>>>> >>>>>
>>>>>>>> >>>>> This kind of compile-time environment could conceivably be
>>>>>>>> exposed to
>>>>>>>> >>>>> macros. Common Lisp had proposals along that line
>>>>>>>> >>>>> (https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node102.html)
>>>>>>>> but as far as
>>>>>>>> >>>>> I can tell, it was too complicated and not useful enough, so
>>>>>>>> it was
>>>>>>>> >>>>> axed/neutered at some point in the standardization process.
>>>>>>>> >>>>>
>>>>>>>> >>>>> > Hadley Wickham's lazyeval package in R is pretty cool in
>>>>>>>> that you can
>>>>>>>> >>>>> > attach an environment to an expression, pass it in and out
>>>>>>>> of functions with
>>>>>>>> >>>>> > various modifications, and then evaluate the expression
>>>>>>>> within the original
>>>>>>>> >>>>> > environment
>>>>>>>> >>>>>
>>>>>>>> >>>>> I don't know about R, but to me that sounds entirely doable
>>>>>>>> with
>>>>>>>> >>>>> closures (and macros will give you a nice syntax for it)
>>>>>>>> >>>>>
>>>>>>>> >>>>>>
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> On Wednesday, July 8, 2015 at 8:31:44 PM UTC-4, Yichao Yu
>>>>>>>> wrote:
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> On Wed, Jul 8, 2015 at 8:23 PM, Yichao Yu <yyc...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> >>>>>>> > On Wed, Jul 8, 2015 at 7:48 PM, Brandon Taylor
>>>>>>>> >>>>>>> > <brandon....@gmail.com> wrote:
>>>>>>>> >>>>>>> >> Hmm, maybe I'm confused about compilation vs
>>>>>>>> interpretation. Let
>>>>>>>> >>>>>>> >> me
>>>>>>>> >>>>>>> >> rephrase. Regardless of a how or when statement is
>>>>>>>> evaluated, it
>>>>>>>> >>>>>>> >> must have
>>>>>>>> >>>>>>> >> access at least to its parent environments to
>>>>>>>> successfully resolve
>>>>>>>> >>>>>>> >> a symbol.
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> AFAIK, the only scope you can dynamically add variable to
>>>>>>>> is the
>>>>>>>> >>>>>>> global scope. (This can be done with the `global` keyword
>>>>>>>> or `eval`
>>>>>>>> >>>>>>> etc). The compiler figure out the creation/lifetime of all
>>>>>>>> local
>>>>>>>> >>>>>>> variables (at compile time). Therefore, to access a
>>>>>>>> variable in the
>>>>>>>> >>>>>>> parent scope:
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> 1. If it's a global, then it need a runtime lookup/binding
>>>>>>>> (the
>>>>>>>> >>>>>>> reason
>>>>>>>> >>>>>>> global are slow)
>>>>>>>> >>>>>>> 2. If it's in a parent non-global scope, the compiler can
>>>>>>>> figure out
>>>>>>>> >>>>>>> how to bind/access it at compile time and no extra (lookup)
>>>>>>>> code at
>>>>>>>> >>>>>>> runtime is necessary.
>>>>>>>> >>>>>>>
>>>>>>>> >>>>>>> >>
>>>>>>>> >>>>>>> >
>>>>>>>> >>>>>>> > A julia local variable is basically a variable in C.
>>>>>>>> There's a
>>>>>>>> >>>>>>> > table
>>>>>>>> >>>>>>> > at compile time to map between symbols and stack slots
>>>>>>>> (or
>>>>>>>> >>>>>>> > whereever
>>>>>>>> >>>>>>> > they are stored) but such a map does not exist at runtime
>>>>>>>> anymore
>>>>>>>> >>>>>>> > (except for debugging).
>>>>>>>> >>>>>>> >
>>>>>>>> >>>>>>> >>
>>>>>>>> >>>>>>> >> On Wednesday, July 8, 2015 at 7:34:09 PM UTC-4, Brandon
>>>>>>>> Taylor
>>>>>>>> >>>>>>> >> wrote:
>>>>>>>> >>>>>>> >>>
>>>>>>>> >>>>>>> >>> They must exist at runtime and at local scope.
>>>>>>>> Evaluating a
>>>>>>>> >>>>>>> >>> symbol is
>>>>>>>> >>>>>>> >>> impossible without a pool of defined symbols in various
>>>>>>>> scopes to
>>>>>>>> >>>>>>> >>> match it
>>>>>>>> >>>>>>> >>> to. Unless I'm missing something?
>>>>>>>> >>>>>>> >>>
>>>>>>>> >>>>>>> >>> On Wednesday, July 8, 2015 at 7:26:27 PM UTC-4, Jameson
>>>>>>>> wrote:
>>>>>>>> >>>>>>> >>>>
>>>>>>>> >>>>>>> >>>> There are global symbol tables for static analysis /
>>>>>>>> reflection,
>>>>>>>> >>>>>>> >>>> but they
>>>>>>>> >>>>>>> >>>> do not exist at runtime or for the local scope.
>>>>>>>> >>>>>>> >>>>
>>>>>>>> >>>>>>> >>>> On Wed, Jul 8, 2015 at 7:06 PM Brandon Taylor
>>>>>>>> >>>>>>> >>>> <brandon....@gmail.com>
>>>>>>>> >>>>>>> >>>> wrote:
>>>>>>>> >>>>>>> >>>>>
>>>>>>>> >>>>>>> >>>>> Surely environments already exist somewhere inside
>>>>>>>> Julia? How
>>>>>>>> >>>>>>> >>>>> else could
>>>>>>>> >>>>>>> >>>>> you keep track of scope? It would be simply a matter
>>>>>>>> of
>>>>>>>> >>>>>>> >>>>> granting users
>>>>>>>> >>>>>>> >>>>> access to them. Symbol tables in a mutable language
>>>>>>>> are by
>>>>>>>> >>>>>>> >>>>> default mutable.
>>>>>>>> >>>>>>> >>>>> It would certainly be possible only give users access
>>>>>>>> to
>>>>>>>> >>>>>>> >>>>> immutable
>>>>>>>> >>>>>>> >>>>> reifications (which could solve a bunch of problems
>>>>>>>> as is).
>>>>>>>> >>>>>>> >>>>> However, it
>>>>>>>> >>>>>>> >>>>> seems natural to match mutable symbol tables with
>>>>>>>> mutable
>>>>>>>> >>>>>>> >>>>> reifications, and
>>>>>>>> >>>>>>> >>>>> immutable symbol tables with immutable reifications.
>>>>>>>> >>>>>>> >>>>>
>>>>>>>> >>>>>>> >>>>>
>>>>>>>> >>>>>>> >>>>> On Wednesday, July 8, 2015 at 6:50:03 PM UTC-4,
>>>>>>>> Brandon Taylor
>>>>>>>> >>>>>>> >>>>> wrote:
>>>>>>>> >>>>>>> >>>>>>
>>>>>>>> >>>>>>> >>>>>> I'm not sure I understand...
>>>>>>>> >>>>>>> >>>>>>
>>>>>>>> >>>>>>> >>>>>> On Wednesday, July 8, 2015 at 6:24:37 PM UTC-4, John
>>>>>>>> Myles
>>>>>>>> >>>>>>> >>>>>> White wrote:
>>>>>>>> >>>>>>> >>>>>>>
>>>>>>>> >>>>>>> >>>>>>> Reified scope makes static analysis much too hard.
>>>>>>>> Take any
>>>>>>>> >>>>>>> >>>>>>> criticism
>>>>>>>> >>>>>>> >>>>>>> of mutable state: they all apply to globally
>>>>>>>> mutable symbol
>>>>>>>> >>>>>>> >>>>>>> tables.
>>>>>>>> >>>>>>> >>>>>>>
>>>>>>>> >>>>>>> >>>>>>> On Wednesday, July 8, 2015 at 10:26:23 PM UTC+2,
>>>>>>>> Milan
>>>>>>>> >>>>>>> >>>>>>> Bouchet-Valat
>>>>>>>> >>>>>>> >>>>>>> wrote:
>>>>>>>> >>>>>>> >>>>>>>>
>>>>>>>> >>>>>>> >>>>>>>> Le mercredi 08 juillet 2015 à 13:20 -0700, Brandon
>>>>>>>> Taylor a
>>>>>>>> >>>>>>> >>>>>>>> écrit :
>>>>>>>> >>>>>>> >>>>>>>> > All functions.
>>>>>>>> >>>>>>> >>>>>>>> Well, I don't know of any language which doesn't
>>>>>>>> have
>>>>>>>> >>>>>>> >>>>>>>> scoping
>>>>>>>> >>>>>>> >>>>>>>> rules...
>>>>>>>> >>>>>>> >>>>>>>>
>>>>>>>> >>>>>>> >>>>>>>> Anyway, I didn't say scoping rules are necessarily
>>>>>>>> >>>>>>> >>>>>>>> confusing, I was
>>>>>>>> >>>>>>> >>>>>>>> only referring to R formulas. But according to the
>>>>>>>> examples
>>>>>>>> >>>>>>> >>>>>>>> you
>>>>>>>> >>>>>>> >>>>>>>> posted,
>>>>>>>> >>>>>>> >>>>>>>> your question appears to be different.
>>>>>>>> >>>>>>> >>>>>>>>
>>>>>>>> >>>>>>> >>>>>>>>
>>>>>>>>
>>>>>>>
