HIGH CPU Utilization in ScriptSource Compile and 64 Bit mode

May 10, 2010 at 8:25 PM

We are running a web application where we are getting such high CPU usage when compiling that the incoming requests to the IIS start stacking because the Compile causes 100% for 4-8 seconds.  The weird part is that the same code run in 32 bit mode does cause 100% cpu but in a subsecond interval.

Anyone have any idea what may be causing this?  It is using the Iron Python 2.0.1

 

Code as follows:

  var source = _scriptEngine.CreateScriptSourceFromString(currentData.Code, SourceCodeKind.Statements);

  var code = source.Compile();

 

We are running Server 2008 R2

Processor Type: Intel Xeon L5420 @ 2.50GHz
Processor Cores: 8
Processor Speed: 2500 MHz

Memory:

8 GB

Coordinator
May 10, 2010 at 9:16 PM

Can you compile once and save the code for future invocations against different script scopes?  This is the ideal way to use IronPython in the web server.  Also in IronPython 2.6 we have an interpreter which great reduces the amount of CPU required to compile code for each execution.

From: sstevenson [mailto:notifications@codeplex.com]
Sent: Monday, May 10, 2010 12:26 PM
To: Dino Viehland
Subject: HIGH CPU Utilization in ScriptSource Compile and 64 Bit mode [dlr:212185]

From: sstevenson

We are running a web application where we are getting such high CPU usage when compiling that the incoming requests to the IIS start stacking because the Compile causes 100% for 4-8 seconds. The weird part is that the same code run in 32 bit mode does cause 100% cpu but in a subsecond interval.

Anyone have any idea what may be causing this? It is using the Iron Python 2.0.1

Code as follows:

var source = _scriptEngine.CreateScriptSourceFromString(currentData.Code, SourceCodeKind.Statements);

var code = source.Compile();

We are running Server 2008 R2

Processor Type:

Intel Xeon L5420 @ 2.50GHz

Processor Cores:

8

Processor Speed:

2500 MHz

Memory:

8 GB

Read the full discussion online.

To add a post to this discussion, reply to this email (dlr@discussions.codeplex.com)

To start a new discussion for this project, email dlr@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com

May 10, 2010 at 9:23 PM

The embedded interpreter might just be the solution to this issue as it behaves like that is where the problem centers.   But to answer your question yes, we only compile this code once if the data or script is updated.  But that once is enough to bring the site to a bogged down state.  This code runs on a timer and only compiles if the script had changed.

 

private void DoCheckAndCompile()
        {
            var cachedData = _cachedData;
            var cachedCode = _cachedCode;
            var currentData = _rulesStore.Data;

            if (AreEqual(currentData, cachedData) && cachedCode != null)
                return;

            var cacheLock = new object();
            if (Monitor.TryEnter(cacheLock))
            {

                try
                {
                    var source = _scriptEngine.CreateScriptSourceFromString(currentData.Code, SourceCodeKind.Statements);
                    var code = source.Compile();

                    _cachedCode = code;
                    _cachedData = currentData;   
                }
                finally
                {
                    Monitor.Exit(cacheLock);
                }
            }
        }

Coordinator
May 10, 2010 at 9:27 PM

Could you compile the code on a background thread and pick up the newly compiled code once it becomes available?  Unfortunately the 64-bit JIT is known to be slower than the 32-bit JIT and there’s not a whole lot we can do about that (other than all the work we’ve done so far and continue to work on to avoid JITing – 2.6 is definitely much, much better in this regard both here and in other areas).

From: sstevenson [mailto:notifications@codeplex.com]
Sent: Monday, May 10, 2010 1:24 PM
To: Dino Viehland
Subject: Re: HIGH CPU Utilization in ScriptSource Compile and 64 Bit mode [dlr:212185]

From: sstevenson

The embedded interpreter might just be the solution to this issue as it behaves like that is where the problem centers. But to answer your question yes, we only compile this code once if the data or script is updated. But that once is enough to bring the site to a bogged down state. This code runs on a timer and only compiles if the script had changed.

private void DoCheckAndCompile()
{
var cachedData = _cachedData;
var cachedCode = _cachedCode;
var currentData = _rulesStore.Data;

if (AreEqual(currentData, cachedData) && cachedCode != null)
return;

var cacheLock = new object();
if (Monitor.TryEnter(cacheLock))
{

try
{
var source = _scriptEngine.CreateScriptSourceFromString(currentData.Code, SourceCodeKind.Statements);
var code = source.Compile();

_cachedCode = code;
_cachedData = currentData;
}
finally
{
Monitor.Exit(cacheLock);
}
}
}

Read the full discussion online.

To add a post to this discussion, reply to this email (dlr@discussions.codeplex.com)

To start a new discussion for this project, email dlr@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com

May 10, 2010 at 10:46 PM

If I am not mistaken the timer is on its own thread.  I am going to definitely try the new version.  Looks like some items in microsoft.scripting are no longer there that we were using.  ExtensionType is no longer in that assembly so have to dig that up.

May 10, 2010 at 10:48 PM

They removed it from microsoft.scripting.dl and I found it in microsoft.dynamic.dll

Coordinator
May 10, 2010 at 10:49 PM

Ahh yes – lots of stuff moved to Microsoft.Dynamic.dll from Microsoft.Scripting.dll – we separated the hosting APIs out into their own stand alone DLL from everything else in the DLR.

From: sstevenson [mailto:notifications@codeplex.com]
Sent: Monday, May 10, 2010 2:48 PM
To: Dino Viehland
Subject: Re: HIGH CPU Utilization in ScriptSource Compile and 64 Bit mode [dlr:212185]

From: sstevenson

They removed it from microsoft.scripting.dl and I found it in microsoft.dynamic.dll

Read the full discussion online.

To add a post to this discussion, reply to this email (dlr@discussions.codeplex.com)

To start a new discussion for this project, email dlr@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com

May 20, 2010 at 8:43 PM

Okay I tried to upgrade to 2.6.x and now I get stack overflow after throwing traffic at the site in 64 bit mode.  I have attached the debugger and made a dump with the breakpoint at the point the exception is raised.   Any idea what changed between 2.0 and 2.6 that might cause StackOverflow? 

Coordinator
May 20, 2010 at 8:44 PM

Can you send the stack trace?  Lots has changed between 2.0 and 2.6 so it’s hard to pin down what it could be w/o it.

From: sstevenson [mailto:notifications@codeplex.com]
Sent: Thursday, May 20, 2010 12:43 PM
To: Dino Viehland
Subject: Re: HIGH CPU Utilization in ScriptSource Compile and 64 Bit mode [dlr:212185]

From: sstevenson

Okay I tried to upgrade to 2.6.x and now I get stack overflow after throwing traffic at the site in 64 bit mode. I have attached the debugger and made a dump with the breakpoint at the point the exception is raised. Any idea what changed between 2.0 and 2.6 that might cause StackOverflow?

Read the full discussion online.

To add a post to this discussion, reply to this email (dlr@discussions.codeplex.com)

To start a new discussion for this project, email dlr@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com

May 20, 2010 at 11:02 PM

Okay we did some varying tests today and we found that its happening when there is a large number of ElsIf Statements in Python.  Once we would exceed 125 of them the stack overflows would start happening.

OS Thread Id: 0x1788 (37)
Child-SP         RetAddr          Call Site
00000000095564d0 000007fef9ca12e6 KERNELBASE!RaiseException+0x39
00000000095565a0 000007fef9ecfdd5 mscorwks!`string'+0x59046
00000000095566f0 000007fef8baa137 mscorwks!RuntimeTypeHandle::GetInstantiation+0x175
00000000095568f0 000007ff009d5750 mscorlib_ni!System.RuntimeType.GetGenericArguments()+0x37
0000000009556950 000007ff009d5351 Microsoft_Scripting_Core!System.Dynamic.Utils.TypeUtils.CanCache(System.Type)+0x60
0000000009556990 000007ff00b11276 Microsoft_Scripting_Core!System.Dynamic.Utils.TypeExtensions.GetParametersCached(System.Reflection.MethodBase)+0x71
00000000095569f0 000007ff00b11115 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Expression.GetValidMethodForDynamic(System.Type)+0x26
0000000009556a30 000007ff00c17f15 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Expression.MakeDynamic(System.Type, System.Runtime.CompilerServices.CallSiteBinder, Microsoft.Scripting.Ast.Expression, Microsoft.Scripting.Ast.Expression)+0x75
0000000009556aa0 000007ff00c171ca Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteDynamicExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xe5
0000000009556b10 000007ff00c180cd Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xd2a
0000000009556f70 000007ff00c1803f Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller+ChildRewriter.Add(Microsoft.Scripting.Ast.Expression)+0x4d
0000000009556fc0 000007ff00c17eb8 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller+ChildRewriter.AddArguments(Microsoft.Scripting.Ast.IArgumentProvider)+0x4f
0000000009557010 000007ff00c171ca Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteDynamicExpression(Microsoft.Scripting.Ast.Expression, Stack)+0x88
0000000009557080 000007ff00c180cd Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xd2a
00000000095574e0 000007ff00c1803f Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller+ChildRewriter.Add(Microsoft.Scripting.Ast.Expression)+0x4d
0000000009557530 000007ff00c17eb8 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller+ChildRewriter.AddArguments(Microsoft.Scripting.Ast.IArgumentProvider)+0x4f
0000000009557580 000007ff00c171ca Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteDynamicExpression(Microsoft.Scripting.Ast.Expression, Stack)+0x88
00000000095575f0 000007ff00c17993 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xd2a
0000000009557a50 000007ff00c1691a Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteConditionalExpression(Microsoft.Scripting.Ast.Expression, Stack)+0x93
0000000009557b20 000007ff00c17803 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0x47a
0000000009557f80 000007ff00c17156 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteBlockExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xa3
0000000009558030 000007ff00c179f9 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xcb6

last 4 lines Repeated 122 times


00000000095aa980 000007ff00c1691a Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteConditionalExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xf9
00000000095aaa50 000007ff00c17803 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0x47a
00000000095aaeb0 000007ff00c17156 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteBlockExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xa3
00000000095aaf60 000007ff00c179f9 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xcb6
00000000095ab3c0 000007ff00c1691a Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteConditionalExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xf9
00000000095ab490 000007ff00c17803 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0x47a
00000000095ab8f0 000007ff00c17156 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteBlockExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xa3
00000000095ab9a0 000007ff00c17803 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xcb6
00000000095abe00 000007ff00c17156 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteBlockExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xa3
00000000095abeb0 000007ff012b14af Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xcb6
00000000095ac310 000007ff00c17206 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExtensionExpression(Microsoft.Scripting.Ast.Expression, Stack)+0x6f
00000000095ac390 000007ff00c17803 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xd66
00000000095ac7f0 000007ff00c17156 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteBlockExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xa3
00000000095ac8a0 000007ff012b10f6 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xcb6
00000000095acd00 000007ff00c173da Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteTryExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xe6
00000000095ace50 000007ff00c17803 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xf3a
00000000095ad2b0 000007ff00c17156 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteBlockExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xa3
00000000095ad360 000007ff00c17803 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xcb6
00000000095ad7c0 000007ff00c17156 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteBlockExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xa3
00000000095ad870 000007ff00c18453 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xcb6
00000000095adcd0 000007ff00c172f0 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteLabelExpression(Microsoft.Scripting.Ast.Expression, Stack)+0x73
00000000095add70 000007ff00c163e8 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xe50
00000000095ae1d0 000007ff00c16245 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpressionFreeTemps(Microsoft.Scripting.Ast.Expression, Stack)+0x48
00000000095ae240 000007ff00c161d5 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.Rewrite[[System.__Canon, mscorlib]](Microsoft.Scripting.Ast.Expression`1<System.__Canon>)+0x45
00000000095ae2d0 000007ff00c16014 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Expression`1[[System.__Canon, mscorlib]].Accept(Microsoft.Scripting.Ast.Compiler.StackSpiller)+0x55
00000000095ae320 000007ff012b0355 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.LambdaCompiler.Compile(Microsoft.Scripting.Ast.LambdaExpression, System.Runtime.CompilerServices.DebugInfoGenerator)+0x44
00000000095ae370 000007fef8b3dd38 Microsoft_Dynamic!Microsoft.Scripting.Interpreter.LightDelegateCreator.Compile(System.Object)+0xb5
00000000095ae3d0 000007fef9aad502 mscorlib_ni!System.Threading.ExecutionContext.runTryCode(System.Object)+0x178
00000000095ae490 000007fef9969fd3 mscorwks!CallDescrWorker+0x82
00000000095ae4e0 000007fef997a3af mscorwks!CallDescrWorkerWithHandler+0xd3
00000000095ae580 000007fef995f916 mscorwks!MethodDesc::CallDescr+0x24f
00000000095ae7e0 000007fef9f0a302 mscorwks!ExecuteCodeWithGuaranteedCleanupHelper+0x12a
00000000095aea70 000007fef8b22b82 mscorwks!ReflectionInvocation::ExecuteCodeWithGuaranteedCleanup+0x172
00000000095aec80 000007fef8b97411 mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+0x62
00000000095aecd0 000007fef8b9724f mscorlib_ni!System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(System.Threading._ThreadPoolWaitCallback)+0x61
00000000095aed20 000007fef9aad502 mscorlib_ni!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(System.Object)+0x4f
00000000095aed70 000007fef9969fd3 mscorwks!CallDescrWorker+0x82
00000000095aedc0 000007fef9961c3a mscorwks!CallDescrWorkerWithHandler+0xd3
00000000095aee60 000007fef991d0bf mscorwks!DispatchCallDebuggerWrapper+0x3e
00000000095aeec0 000007fef9a22f17 mscorwks!DispatchCallNoEH+0x5f
00000000095aef40 000007fef98ec1a0 mscorwks!QueueUserWorkItemManagedCallback+0x83
00000000095aefd0 000007fef9928725 mscorwks!Thread::DoADCallBack+0x488
00000000095af020 000007fef98e0195 mscorwks!SVR::gc_heap::make_heap_segment+0x155
00000000095af0f0 000007fef98c0a21 mscorwks!AssemblySecurityDescriptor::GetZone+0x169
00000000095af130 000007fef98ebe5d mscorwks!ClassCompat::MethodTableBuilder::BuildInteropVTable_PlaceVtableMethods+0x441
00000000095af160 000007fef98ec1c5 mscorwks!Thread::DoADCallBack+0x145
00000000095af2d0 000007fef9928725 mscorwks!Thread::DoADCallBack+0x4ad
00000000095af320 000007fef98e0195 mscorwks!SVR::gc_heap::make_heap_segment+0x155
00000000095af3f0 000007fef98c4cc5 mscorwks!AssemblySecurityDescriptor::GetZone+0x169
00000000095af430 000007fef98f102a mscorwks!ThreadNative::KickOffThread+0x401
00000000095af490 000007fef99ca4ea mscorwks!ManagedPerAppDomainTPCount::DispatchWorkItem+0xee
00000000095af540 000007fef9a2095c mscorwks!ThreadpoolMgr::WorkerThreadStart+0x1ba
00000000095af5e0 00000000775cf56d mscorwks!Thread::intermediateThreadProc+0x78
00000000095af7b0 0000000077703281 kernel32!BaseThreadInitThunk+0xd
00000000095af7e0 0000000000000000 ntdll!RtlUserThreadStart+0x1d

Coordinator
May 20, 2010 at 11:28 PM

Is it 123 if/else blocks?  I’ll open a bug on this and we can probably add a workaround for IronPython 2.6.2 or 2.6.3 where we generate flatter trees w/ gotos from the IronPython side when we have a large number of if/elses.  Unfortunately the real fix would be not using these recursive algorithms in the DLR to re-write trees and compile code but that would be a very significant change and is probably way out on the horizon. 

One possible workaround might be to only ever use the interpreter by setting the compilation threshold very high (using –X:CompilationTheshold option or setting the option of the same name when setting up the script runtime).  That’d have a significant throughput hit though.   Other than that I can only recommend changing the code w/ the if/else blocks for the time being L 

From: sstevenson [mailto:notifications@codeplex.com]
Sent: Thursday, May 20, 2010 3:02 PM
To: Dino Viehland
Subject: Re: HIGH CPU Utilization in ScriptSource Compile and 64 Bit mode [dlr:212185]

From: sstevenson

Okay we did some varying tests today and we found that its happening when there is a large number of ElsIf Statements in Python. Once we would exceed 125 of them the stack overflows would start happening.

OS Thread Id: 0x1788 (37)
Child-SP RetAddr Call Site
00000000095564d0 000007fef9ca12e6 KERNELBASE!RaiseException+0x39
00000000095565a0 000007fef9ecfdd5 mscorwks!`string'+0x59046
00000000095566f0 000007fef8baa137 mscorwks!RuntimeTypeHandle::GetInstantiation+0x175
00000000095568f0 000007ff009d5750 mscorlib_ni!System.RuntimeType.GetGenericArguments()+0x37
0000000009556950 000007ff009d5351 Microsoft_Scripting_Core!System.Dynamic.Utils.TypeUtils.CanCache(System.Type)+0x60
0000000009556990 000007ff00b11276 Microsoft_Scripting_Core!System.Dynamic.Utils.TypeExtensions.GetParametersCached(System.Reflection.MethodBase)+0x71
00000000095569f0 000007ff00b11115 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Expression.GetValidMethodForDynamic(System.Type)+0x26
0000000009556a30 000007ff00c17f15 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Expression.MakeDynamic(System.Type, System.Runtime.CompilerServices.CallSiteBinder, Microsoft.Scripting.Ast.Expression, Microsoft.Scripting.Ast.Expression)+0x75
0000000009556aa0 000007ff00c171ca Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteDynamicExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xe5
0000000009556b10 000007ff00c180cd Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xd2a
0000000009556f70 000007ff00c1803f Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller+ChildRewriter.Add(Microsoft.Scripting.Ast.Expression)+0x4d
0000000009556fc0 000007ff00c17eb8 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller+ChildRewriter.AddArguments(Microsoft.Scripting.Ast.IArgumentProvider)+0x4f
0000000009557010 000007ff00c171ca Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteDynamicExpression(Microsoft.Scripting.Ast.Expression, Stack)+0x88
0000000009557080 000007ff00c180cd Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xd2a
00000000095574e0 000007ff00c1803f Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller+ChildRewriter.Add(Microsoft.Scripting.Ast.Expression)+0x4d
0000000009557530 000007ff00c17eb8 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller+ChildRewriter.AddArguments(Microsoft.Scripting.Ast.IArgumentProvider)+0x4f
0000000009557580 000007ff00c171ca Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteDynamicExpression(Microsoft.Scripting.Ast.Expression, Stack)+0x88
00000000095575f0 000007ff00c17993 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xd2a
0000000009557a50 000007ff00c1691a Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteConditionalExpression(Microsoft.Scripting.Ast.Expression, Stack)+0x93
0000000009557b20 000007ff00c17803 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0x47a
0000000009557f80 000007ff00c17156 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteBlockExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xa3
0000000009558030 000007ff00c179f9 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xcb6

last 4 lines Repeated 122 times


00000000095aa980 000007ff00c1691a Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteConditionalExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xf9
00000000095aaa50 000007ff00c17803 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0x47a
00000000095aaeb0 000007ff00c17156 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteBlockExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xa3
00000000095aaf60 000007ff00c179f9 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xcb6
00000000095ab3c0 000007ff00c1691a Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteConditionalExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xf9
00000000095ab490 000007ff00c17803 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0x47a
00000000095ab8f0 000007ff00c17156 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteBlockExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xa3
00000000095ab9a0 000007ff00c17803 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xcb6
00000000095abe00 000007ff00c17156 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteBlockExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xa3
00000000095abeb0 000007ff012b14af Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xcb6
00000000095ac310 000007ff00c17206 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExtensionExpression(Microsoft.Scripting.Ast.Expression, Stack)+0x6f
00000000095ac390 000007ff00c17803 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xd66
00000000095ac7f0 000007ff00c17156 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteBlockExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xa3
00000000095ac8a0 000007ff012b10f6 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xcb6
00000000095acd00 000007ff00c173da Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteTryExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xe6
00000000095ace50 000007ff00c17803 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xf3a
00000000095ad2b0 000007ff00c17156 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteBlockExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xa3
00000000095ad360 000007ff00c17803 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xcb6
00000000095ad7c0 000007ff00c17156 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteBlockExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xa3
00000000095ad870 000007ff00c18453 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xcb6
00000000095adcd0 000007ff00c172f0 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteLabelExpression(Microsoft.Scripting.Ast.Expression, Stack)+0x73
00000000095add70 000007ff00c163e8 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpression(Microsoft.Scripting.Ast.Expression, Stack)+0xe50
00000000095ae1d0 000007ff00c16245 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.RewriteExpressionFreeTemps(Microsoft.Scripting.Ast.Expression, Stack)+0x48
00000000095ae240 000007ff00c161d5 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.StackSpiller.Rewrite[[System.__Canon, mscorlib]](Microsoft.Scripting.Ast.Expression`1<System.__Canon>)+0x45
00000000095ae2d0 000007ff00c16014 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Expression`1[[System.__Canon, mscorlib]].Accept(Microsoft.Scripting.Ast.Compiler.StackSpiller)+0x55
00000000095ae320 000007ff012b0355 Microsoft_Scripting_Core!Microsoft.Scripting.Ast.Compiler.LambdaCompiler.Compile(Microsoft.Scripting.Ast.LambdaExpression, System.Runtime.CompilerServices.DebugInfoGenerator)+0x44
00000000095ae370 000007fef8b3dd38 Microsoft_Dynamic!Microsoft.Scripting.Interpreter.LightDelegateCreator.Compile(System.Object)+0xb5
00000000095ae3d0 000007fef9aad502 mscorlib_ni!System.Threading.ExecutionContext.runTryCode(System.Object)+0x178
00000000095ae490 000007fef9969fd3 mscorwks!CallDescrWorker+0x82
00000000095ae4e0 000007fef997a3af mscorwks!CallDescrWorkerWithHandler+0xd3
00000000095ae580 000007fef995f916 mscorwks!MethodDesc::CallDescr+0x24f
00000000095ae7e0 000007fef9f0a302 mscorwks!ExecuteCodeWithGuaranteedCleanupHelper+0x12a
00000000095aea70 000007fef8b22b82 mscorwks!ReflectionInvocation::ExecuteCodeWithGuaranteedCleanup+0x172
00000000095aec80 000007fef8b97411 mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+0x62
00000000095aecd0 000007fef8b9724f mscorlib_ni!System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(System.Threading._ThreadPoolWaitCallback)+0x61
00000000095aed20 000007fef9aad502 mscorlib_ni!System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(System.Object)+0x4f
00000000095aed70 000007fef9969fd3 mscorwks!CallDescrWorker+0x82
00000000095aedc0 000007fef9961c3a mscorwks!CallDescrWorkerWithHandler+0xd3
00000000095aee60 000007fef991d0bf mscorwks!DispatchCallDebuggerWrapper+0x3e
00000000095aeec0 000007fef9a22f17 mscorwks!DispatchCallNoEH+0x5f
00000000095aef40 000007fef98ec1a0 mscorwks!QueueUserWorkItemManagedCallback+0x83
00000000095aefd0 000007fef9928725 mscorwks!Thread::DoADCallBack+0x488
00000000095af020 000007fef98e0195 mscorwks!SVR::gc_heap::make_heap_segment+0x155
00000000095af0f0 000007fef98c0a21 mscorwks!AssemblySecurityDescriptor::GetZone+0x169
00000000095af130 000007fef98ebe5d mscorwks!ClassCompat::MethodTableBuilder::BuildInteropVTable_PlaceVtableMethods+0x441
00000000095af160 000007fef98ec1c5 mscorwks!Thread::DoADCallBack+0x145
00000000095af2d0 000007fef9928725 mscorwks!Thread::DoADCallBack+0x4ad
00000000095af320 000007fef98e0195 mscorwks!SVR::gc_heap::make_heap_segment+0x155
00000000095af3f0 000007fef98c4cc5 mscorwks!AssemblySecurityDescriptor::GetZone+0x169
00000000095af430 000007fef98f102a mscorwks!ThreadNative::KickOffThread+0x401
00000000095af490 000007fef99ca4ea mscorwks!ManagedPerAppDomainTPCount::DispatchWorkItem+0xee
00000000095af540 000007fef9a2095c mscorwks!ThreadpoolMgr::WorkerThreadStart+0x1ba
00000000095af5e0 00000000775cf56d mscorwks!Thread::intermediateThreadProc+0x78
00000000095af7b0 0000000077703281 kernel32!BaseThreadInitThunk+0xd
00000000095af7e0 0000000000000000 ntdll!RtlUserThreadStart+0x1d

Read the full discussion online.

To add a post to this discussion, reply to this email (dlr@discussions.codeplex.com)

To start a new discussion for this project, email dlr@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com

May 21, 2010 at 4:08 PM
Edited May 21, 2010 at 4:35 PM

We counted and it ended up being a 125 if/else block that would bring it in from the stackoverflow.   When you need to replicated it we are running in 64 bit mode using server 2008 R2 IIS 7.5.7600.16385.  Basically we have the script compiled on 1 minute intervals only if it changes but the pages load and use the scripts on each load.   I noticed hitting it with 1 thread with TinyGet it would take longer to error out, once you had simultaneous requests it would Throw the exception almost immediately after starting the requests.  We will investigate other ways to incorporate the If/Else blocks to try to eliminate the recursion.

May 21, 2010 at 4:13 PM

Question I have on this also, why is this code being executed on requests?  It looks like it could possibly be in the compile of the python script.   Or is these methods called for every request?

Coordinator
May 21, 2010 at 5:41 PM

What happens in 2.6 is we start off interpreting your code and run it that way until it’s been called a certain number of times. Once it’s crossed a threshold we begin compiling the code on a background thread – on the stack you’ll see QueueUserWorkItemManagedCallback on the stack which is because we’re on a threadpool thread rather than the thread which is handling the request. Meanwhile we continue to use the interpreted version for any calls. Once the compilation is complete we’ll start using the compiled version of the method instead of the interpreted version. That gives us a good balance between fast startup time and good throughput performance. It’s also why raising the compilation threshold would probably solve your stack overflow problem but give you a new throughput performance problem.

From: sstevenson [mailto:notifications@codeplex.com]
Sent: Friday, May 21, 2010 8:14 AM
To: Dino Viehland
Subject: Re: HIGH CPU Utilization in ScriptSource Compile and 64 Bit mode [dlr:212185]

From: sstevenson

Question I have on this also, why is this code being executed on requests? It looks like it could possibly be in the compile of the python script. Or is these methods called for every request?

Read the full discussion online.

To add a post to this discussion, reply to this email (dlr@discussions.codeplex.com)

To start a new discussion for this project, email dlr@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe or change your settings on codePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at codeplex.com

May 24, 2010 at 5:01 PM

Thanks much for all the help.  If there is anything I can do to help with this particular issue let me know.

Jun 2, 2010 at 4:25 PM

We have now converted to using Dictionary's in the python script to work around the stack overflow issue.    An example would be as follows for a work around.

 

Old code:

if test == "a" :
    zone = "a"
elif test == "b" :
    zone = "b"
elif test == "c" :
    zone = "c"

New Code:

dictPossibilities = {
     "a" : "a"
     "b" : "b"
     "c" : "c"
}
if dictPossibilities.has_key("a") :
     zone = dictPossibilities["a"]

or

def setzone(value) :
     zone = value

def onerr() :
     TODO handle error

dictPossibilities = {
     "a" : setzone("a")
     "b" : setzone("b")
     "c" : setzone("c")
}

dictPossibilities.get("a", onerr)()

 

 

Also note the original CPU issue is definitely resolved in the newer IronPython 2.6.x.