Mixed Method Calling from DLR Languages

Apr 20, 2009 at 6:00 PM
Edited Apr 21, 2009 at 4:33 PM
Hi,

we have powerful mixin support in our open source re-motion .NET enterprise application framework (http://www.re-motion.org/blogs/team/category/4.aspx) and are facing the following ambiguity problem when calling mixed methods from within IronPython: If there exist two mixed methods with the same name extending the same class, then the methods cannot be called without explicitely qualifying them. This seems due to the fact that mixing is done in our implementation using explicit interface implemenations, which in this case leads to an ambiguity from the point of view of DLR languages (in C# you always have to cast to the right interface to access its mixed method(s) so no ambiguity arises).

If an extension developer implements a re-motion extension (consisting of mixins + DLR scripts) the mixed methods are not ambigous by design of the developer and can therefore be called from IPy very elegantly through e.g. foo.MixedBar(args)

However, if the MixedBar method becomes ambigous due to another extension mixing a method of the same name to the same class (something which the extension developer has evidently no control over), all of the methods coming from this mixin can no longer be called directly in IronPython, but need to be fully qualified using a function- intead of a method style call: e.g.
The.Long.Mixin.Namespace.IMyAmbigousMixinInterface1.MixedBar(foo, args)

In the ambigous case executing the script will throw when using using foo.MixedBar(args). This can only be avoided if the extension developer always calls all mixed methods in fully qualified function style !
 
For convenience & consistency reasons we want extension developers to be able to always use the simple & elegant foo.MixedBar(args), without needing to worry about ambiguities with other mixins.  

Any pointers on how we can we achieve this (if possible DLR language agnostic, so we can support DLR languages apart from IPy) to keep Mixins more elegant than C# even in the case of ambiguity, would be great.

Cheers,
Markus


PS: This is unfortunately already a rather long post, but to elaborate a bit more we would like the extension developer be able to use:

# ScriptSource1, coming from extension 1: MixedBar method coming from IMyAmbigousMixinInterface1 shall be called
foo.MixedBar(args)

# ScriptSource2, coming from extension 2: MixedBar method coming from IMyAmbigousMixinInterface2 shall be called
foo.MixedBar(args)

Basically what we want is, that script code coming from extension 1 shall always call IMyAmbigousMixinInterface1.MixedBar, script code coming from extension 2 shall always call IMyAmbigousMixinInterface2.MixedBar .

We create scripts using
var scriptSourceExtension1 = pythonEngine.CreateScriptSourceFromString(scriptTextExtension1, SourceCodeKind.Statements);
But even if we then execute them in different ScriptScope|s
scriptSourceExtension1.Execute (scopeExtension1);
the DLR will still see both mixed methods and the call will still be ambigous.

So it looks like what we would need to achieve is either
a) Prevent the DLR from seeing the "wrong" mixed method when calling CreateScriptSourceFromString
b) Add the correct MixedBar call as a method for the mixed class after the call to CreateScriptSourceFromString
c) Intercept the mechanism that looks for (the non-existent due to ambiguity) MixedBar and subsitute the correct mixed method call during script runtime.

It looks like implementing IDynamicObject would make c) possible, iff we could get the extension context from the Expression passed to GetMetaObject; however in C# 4.0 implementing IDynamicObject will mean that the object will be seen as being dynamic from C# also, which is not something we want for every mixed type.

In any case it would be preferable to get this done at script setup time, which means achieving a) or b); in this case, however, the mixed Foo object on which MixedBar is being called is treated as coming from a statically typed language and there seems to be no way to change the standard  behavior of the type being analyzed by the DLR through reflection and mapped into the dynamic world accordingly.
Coordinator
Apr 20, 2009 at 6:36 PM

What you’re wanting to do here is actually a limitation of IronPython and not the DLR.  IronPython actually gets to choose how it exposes normal .NET types and this is the best approach we’ve come up to for explicit interface implementation (it used to be that you always had to fully qualify them even if there were no conflicts!). 

I think adding a feature to IronPython where sometimes a type is exposed one way and other times it’s exposed another way to IronPython is quite scary.  It’s bound to end up confusing lots of users who get unexpected results depending on how they execute things.  And unfortunately there’s no good way for you to know who is calling your objects if you were to implement IDO so you can’t tell if your object is being called from 1 script or another in a language agnostic way.

Could you instead create proxies around the .NET objects which expose the right members?  You could always have an implicit conversion back to the real type that unwraps it and that’d run if we passed the wrapper off to an API that expects the strongly typed object.

From: MGee [mailto:notifications@codeplex.com]
Sent: Monday, April 20, 2009 10:01 AM
To: Dino Viehland
Subject: Mixed Method Calling from DLR Languages [dlr:53875]

From: MGee

Hi,

we have powerful mixin support in our open source re-motion .NET enterprise application framework (http://www.re-motion.org/blogs/team/category/4.aspx) and are facing the following ambiguity problem when calling mixed methods from within IronPython: If there exist two mixed methods with the same name extending the same class, then the methods cannot be called without explicitely qualifying them. This seems due to the fact that mixing is done in our implementation using explicit interface implemenations, which in this case leads to an ambiguity from the point of view of DLR languages (in C# you always have to cast to the right interface to access its mixed method(s) so no ambiguity arises).

If an extension developer implements a re-motion extension (consisting of mixins + DLR scripts) the mixed methods are not ambigous by design of the developer and can therefore be called from IPy very elegantly through e.g. foo.MixedBar(args)

However, if the MixedBar method becomes ambigous due to another extension mixing a method of the same name to the same class (something which the extension developer has evidently no control over), all of the methods coming from this mixin can no longer be called directly in IronPython, but need to be fully qualified using a function- intead of a method style call: e.g.
The.Long.Mixin.Namespace.IMyAmbigousMixinInterface1.MixedBar(foo, args)

In the ambigous case executing the script will throw when using using foo.MixedBar(args). This can only be avoided if the extension developer always calls all mixed methods in fully qualified function style !

For convenience & consistency reasons we want extension developers to be able to always use the simple & elegant foo.MixedBar(args), without needing to worry about ambiguities with other mixins.

Any pointers on how we can we achieve this (if possible DLR language agnostic, so we can support DLR languages apart from IPy) to keep Mixins more elegant than C# even in the case of ambiguity, would be great.

Cheers,
Markus


PS: This is unfortunately already a rather long post, but to elaborate a bit more we would like the extension developer be able to use:

# ScriptSource1, coming from extension 1: MixedBar method coming from IMyAmbigousMixinInterface1 shall be called
foo.MixedBar(args)

# ScriptSource2, coming from extension 2: MixedBar method coming from IMyAmbigousMixinInterface2 shall be called
foo.MixedBar(args)

Basically what we want is, that script code coming from extension 1 shall always call IMyAmbigousMixinInterface1.MixedBar, script code coming from extension 2 shall always call IMyAmbigousMixinInterface2.MixedBar .

We create scripts using
var scriptSourceExtension1 = pythonEngine.CreateScriptSourceFromString(scriptTextExtension1, SourceCodeKind.Statements);
;
But even if we then execute them in different ScriptScope|s
scriptSourceExtension1.Execute (scopeExtension1);
the DLR will still see both mixed methods and the call will still be ambigous.

So it looks like what we would need to achieve is either
a) Prevent the DLR from seeing the "wrong" mixed method when calling CreateScriptSourceFromString
b) Add the correct MixedBar call as a method for the mixed class after the call to CreateScriptSourceFromString
c) Intercept the mechanism that looks for (the non-existent due to ambiguity) MixedBar and subsitute the correct mixed method call during script runtime.

It looks like implementing IDynamicObject would make c) possible, iff we could get the extension context from the Expression passed to GetMetaObject; however in C# 4.0 implementing IDynamicObject will mean that the object will be seen as being dynamic from C# also, which is not something we want for every mixed type.

In any case it would be preferable to get this done at script setup time, which means achieving a) or b); in this case, however, the mixed Foo object on which MixedBar is being called is treated as coming from a statically typed language and there seems to be no way to change the standard behavior of the type being analyzed by the DLR through reflection and mapped into the dynamic world accordingly.

Read the full discussion online.

To add a post to this discussion, reply to this email (dlr@discussions.codeplex.com)

To start a new discussion for this project, email dlr@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com

Apr 21, 2009 at 4:46 PM

The problem is that for .NET wrapper objects, we’d face the same problem: We don’t know which script the call comes from. These objects would be passed around, and I don’t believe we’d get a chance to switch wrappers between script boundaries.

Just imagine this (admittedly extreme) example:

Script 1 (referencing assembly A):

obj.bar()                  // should invoke bar() as defined in assembly A
foo (lambda x: x.bar())

Script 2 (referencing assembly B):

def foo (method):
    obj.bar()              // should invoke bar() as defined in assembly B
    method (obj)     // should invoke bar() as defined in assembly A, since the lambda expression was created in Script 1.

I don’t see a way to solve this problem without support from the DLR or the script language.

On the other hand, being dependent on a script context does not sound all that scary to me. At least no scarier than watching your carefully tested scripts break in production, and getting error reports from users for unforeseeable situations.

This could be a feature that’s fine to use even without mixins, so I’ll restrict the discussion to explicit interface implementations. In fact, similar mechanisms could even be used for other ambiguity cases.

Consider a script developer who’s writing a script on their own PC, where any number of assemblies is installed. The script developer therefore knows only the types defined in those assemblies. Now we can follow two schools of thought:
a) This is a dynamic language, so whatever the script developer saw while they were developing doesn’t matter.
b) The script developer was scripting for e known environment, and what executed once should execute too in environments that have other assemblies too.
(We could even argue that the script developer should be able to restrict access to a certain set of assemblies that are of interest to them, very much like “add assembly reference” in C#. A call to a method defined in another assembly should then always fail, even if it could unambiguously be resolved on the script developer’s machine.)

Considering any environment that uses DLR languages for application scripting, we have a potential problem of breaking code when 3rd party add-ins are loaded at run time. Option b) would free the script developer from worrying about that, and it would free the admin of the deployed system from manually resolving ambiguities in script source code. This would give us the option to take the best of dynamic languages without the horror of PHP applications, where you end up modifying scripts at deployment time (effectively branching the code).

Now I see that this might seem like a very un-dynamic approach, but it would be of real value to any dynamically extensible system that uses the DLR for scripting in the context of deliverable modules (e.g. workflow activities).

I’m not sure if this is important enough to make it into the DLR or any DLR language (although scripting in large applications with plugin-models seems to be a sweet spot for both), but any mechanisms that would allow us to create this behavior ourselves would be very welcome!

I’ve seen some classes in the DLR source that look like they could provide the necessary context (ScriptDomainManager, LanguageContext, Scope), but they don’t seem to get passed around enough. Or is there a way? Then any hints would be welcome, of course.

What do you think?

- Stefan

Apr 21, 2009 at 6:07 PM
This isn't intrinsically an issue with the DLR. It's very much a matter of language-level semantics, and something that a particular language could certainly choose to implement.  It's just not consistent with what developers expect from Python.

I would do as Dino suggested and create wrappers.  This will have to be exposed directly to users and won't be automatic -- but neither will it be terribly different than performing a typecast in C#.  Here's some sample code that builds an interface-specific wrapper on top of an object.  (I'm sure it could be done more cleanly.)

_wrapper_cache = {}

def Wrapper(interface):
    global _wrapper_cache
    methods = _wrapper_cache.get(interface)
    if methods is None:
        methods = []
        for name in dir(interface):
            member = getattr(interface, name)
            if callable(member):
                methods.append(member)
        _wrapper_cache[interface] = methods
        
    class C(object):
        def __init__(self, obj):
            self._obj = obj
            im = type(C.__init__)
            for method in methods:
                setattr(self, method.__name__, im(method, self._obj))
        def __repr__(self):
            return 'repr(self._obj)
     
    C.__name__ = interface.__name__
    return C

# Sample usage
f = file('test.txt', 'w')
d = Wrapper(System.IDisposable)(f)
d.Dispose()

Apr 22, 2009 at 11:12 AM
Curt,

first, thanks for the code snippet. We might extend this idea to create a wrapper for all unambiguous methods coming from any assembly that is referenced from the script's component. But still, this makes for a rather cumbersome syntax, especially in scenarios where this needs to be applied a lot (i.e., for several objects even in a smallish script).

We also have the problem of not being able to attach specific information to scripts before loading them in a way that they are visible to each other. In my scenario above, that means that foo() would either not be callable from Script 1, or it could not use its own, independent list of referenced assemblies. (Or we would have to manually create this list and reference it in every call that creates a wrapper.)

I agree that what we are trying to do is not what a Python programmer would expect. But if you think about the DLR as a system for embedded scripting (and about IronPython as the most mature scripting language in the DLR universe), it might make sense to think about things from a different perspective. This is not about writing whole apps in Python, this is about extending a large, add-in enabled app with a few scripts, e.g. in workflow steps. We took great care to build an addin-system that does not break when ambiguities arise at deployment time. Static languages make that easy, since they only know what they reference. However, in dynamic languages, adding additional add-ins (what an alliteration!) might create ambiguities at run-time, and this is definately not what any programmer would expect.

We're not asking you to modify the language in any unexpected way, but opening up extensibility points would be just great. We are looking for two specific extensions that would make our lives easier:
I. attaching any information to a script when we load it, so that many modules with different sets of information can co-exist and are visible to/callable from each other (I understand that they need to be in the same ScriptScope)
II. modifying invocation semantics (without implementing IDynamicObject, which we a) do not want to reference while the DLR is beta, and b) would probably not be able to access the information from I.)

Thanks for your help,
Stefan