Wednesday, July 27, 2011

C# Esri Shapefile Reader

I've been trying to populate a Sql Server database with shapefile information from the National Weather Service in an automated fashion, I finally found a tool that helps me do just that.  It's called ESRI Shapefile Reader from CodePlex.

I had first tried Shape2Sql, but it is closed source and for some reason would not import when I had Create Spatial Index checked.  It would run through all the rows in the Shapefile, but no table or rows would be created.  So I pre-created the table, still no rows were created.  That was with the GUI, so when I tried it as a command line tool, again, it would not import the shape file because there is no command line option to disable creating the spatial index.  I don't know where the problem lies, and Shape2Sql appears to be a good tool.  For me, it works great for doing the import by hand, which is fine for data that doesn't change often, but I need automation and if the command line doesn't work, I'm not about to try and manipulate mouse clicks on a GUI from a windows service....

I found various other tools, but they require subscription $, and I am trying to do this with a minimal budget since this is being used for a hobby.

Anyways, I needed to load this data into SqlServer Express.  Using Shape2Sql, it took about 14 minutes for my particular datafile.  I hate waiting and I need to automate this thing, so I decided to use ESRI Shapefile Reader to import a National Weather Service Precipitation file (which can be found at http://water.weather.gov/precip/download.php) and only grab the columns I'm interested in.  Doing this, and threading the import, it now only takes 45 seconds to import the data.  Best of all, I met my goal of doing this in an automated fashion.  Here's my code snippet that shows you how I did this, if you have any questions, just post it.



        public static void CreateWorkForThreads()
        {
            DataSet ds = CreateNewDataSet();
            DataTable dtNWS = ds.Tables[0];

            // Parse the shapefile into a DataTable, grabbing the columns we are interested in
            using (Shapefile shapefile = new Shapefile(Path.Combine(weatherFileDir, "nws_precip_1day_observed_" + dateToLoad.ToString("yyyyMMdd") + ".shp")))
            {
                foreach (Shape shape in shapefile)
                {
                    string[] metadataNames = shape.GetMetadataNames();
                    decimal lat = 0m;
                    decimal lon = 0m;
                    decimal globvalue = 0m;

                    if (metadataNames != null)
                    {
                        foreach (string metadataName in metadataNames)
                        {
                            if (metadataName == "lat")
                                lat = decimal.Parse(shape.GetMetadata(metadataName));
                            else if (metadataName == "lon")
                                lon = decimal.Parse(shape.GetMetadata(metadataName));
                            else if (metadataName == "globvalue")
                                globvalue = decimal.Parse(shape.GetMetadata(metadataName));
                        }
                    }

                    DataRow drNWS = dtNWS.NewRow();
                    drNWS["lat"] = lat;
                    drNWS["lon"] = lon;
                    drNWS["globalvalue"] = globvalue;
                    drNWS["precipDate"] = dateToLoad;
                    drNWS["XAxis"] = Math.Cos(ConvertDegreesToRadians((double)lat)) * Math.Cos(ConvertDegreesToRadians((double)lon));
                    drNWS["YAxis"] = Math.Cos(ConvertDegreesToRadians((double)lat)) * Math.Sin(ConvertDegreesToRadians((double)lon));;
                    drNWS["ZAxis"] = Math.Sin(ConvertDegreesToRadians((double)lat));
                    dtNWS.Rows.Add(drNWS);
                }
            }

            List; listOfDataSetsForThreads = new List();
            DataSet dsCur = CreateNewDataSet();

            // Create a list of datasets, each containing the rows the thread will import
            foreach (DataRow dr in dtNWS.Rows)
            {
                if (dsCur.Tables[0].Rows.Count % 3000 == 0)
                {
                    listOfDataSetsForThreads.Add(dsCur);
                    dsCur = CreateNewDataSet();
                }

                dsCur.Tables[0].ImportRow(dr);
            }

            if (dsCur.Tables[0].Rows.Count > 0)
                listOfDataSetsForThreads.Add(dsCur);

            // Spawn off the threads to import our datasets in parallel
            foreach (DataSet dsThreadWork in listOfDataSetsForThreads)
            {
                WaitCallback wcb = new WaitCallback(ImportDataSet);
                ThreadPool.QueueUserWorkItem(wcb, dsThreadWork);
            }
        }

        public static void ImportDataSet(object o)
        {
            DataSet ds = (DataSet)o;
            using (SqlConnection con = new SqlConnection(ConfigurationManager.AppSettings["myDb"]))
            {
                con.Open();

                try
                {
                    SqlDataAdapter da = new SqlDataAdapter("select top 1 * from nws_precip_history", con);
                    SqlCommandBuilder bldr = new SqlCommandBuilder(da);

                    da.InsertCommand = bldr.GetInsertCommand();
                    da.InsertCommand.UpdatedRowSource = UpdateRowSource.None;
                    da.UpdateBatchSize = 500;
                    da.Update(ds, "nws_precip_history");
                }
                finally
                {
                    if (con.State == ConnectionState.Open)
                        con.Close();
                }
            }

        }

        public static DataSet CreateNewDataSet()
        {
            DataSet dsTemp = new DataSet();
            DataTable dtNWSTemp = new DataTable("nws_precip_history");
            dtNWSTemp.Columns.Add("lat", typeof(decimal));
            dtNWSTemp.Columns.Add("lon", typeof(decimal));
            dtNWSTemp.Columns.Add("globalvalue", typeof(decimal));
            dtNWSTemp.Columns.Add("precipDate", typeof(DateTime));
            dtNWSTemp.Columns.Add("XAxis", typeof(float));
            dtNWSTemp.Columns.Add("YAxis", typeof(float));
            dtNWSTemp.Columns.Add("ZAxis", typeof(float));
            dsTemp.Tables.Add(dtNWSTemp);

            return dsTemp;
        }

        public static double ConvertDegreesToRadians(double degrees)
        {
            double radians = (Math.PI / 180) * degrees;
            return (radians);
        }


Wednesday, July 6, 2011

Failed to access IIS metabase


Failed to access IIS metabase

Problem:
Running an IIS web application, and you get the Failed to access IIS metabase error.

Solution:
First thing to try:
Bring up a command prompt to your .net framework directory under C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727

Run: aspnet_regiis –i

If that doesn’t work, run:  aspnet_regiis -ga aspnet

That should solve it.  If it doesn’t, here’s some reading:  The MetaAcl tool for modifying metabase permissions on the IIS Admin Objects is available

The entry 'AspNetSqlMembershipProvider' has already been added


Problem:

Server Error in '/MyApp' Application.


Configuration Error

Description: An error occurred during the processing of a configuration file required to service this request. Please review the specific error details below and modify your configuration file appropriately.

Parser Error Message: The entry 'AspNetSqlMembershipProvider' has already been added.

Source Error:

Line 42: 
Line 43:   
Line 44:     
Line 45:       enablePasswordRetrieval="false">

Source File: C:\Program Files\MyApp\web.config    Line: 44


Version Information: Microsoft .NET Framework Version:2.0.50727.3620; ASP.NET Version:2.0.50727.3618



Solution:
In the web.config file, you need to add a "remove provider" tag before the add provider tag:



Thursday, June 30, 2011

Login failed for user xxxx. The user is not associated with a trusted SQL Server connection.


Problem:
Connecting to SQL Server with a query in SQL Server Management Studio, you get “Login failed for user xxxx.  The user is not associated with a trusted SQL Server connection.”












Solution:
Open Microsoft SQL Server Management Studio.
Connect to your sql instance.  Right click the instance name in the Object Explorer window and select properties.





Select SQL Server and Windows Authentication mode and click OK.

!Very Important!  Restart SQL Server for this setting to take effect.

Saturday, April 2, 2011

Problem in mapping fragments starting at line nnn:All the key properties (xxxx.column) of the EntitySet xxxx must be mapped to all the key properties

Problem:

Error 3003: Problem in mapping fragments starting at line 257:All the key properties (table.column) of the EntitySet table must be mapped to all the key properties (table.column, table.column) of table table.


Solution:

Select the column giving you problems, hit F4 for properties, and change Nullable from (None) to False.




















Wednesday, March 30, 2011

Index (zero based) must be greater than or equal to zero and less than the size of the argument list

Problem:
System.FormatException was unhandled by user code
  Message=Index (zero based) must be greater than or equal to zero and less than the size of the argument list.
  Source=mscorlib
  StackTrace:
       at System.Text.StringBuilder.AppendFormat(IFormatProvider provider, String format, Object[] args)
       at System.String.Format(IFormatProvider provider, String format, Object[] args)
       at System.String.Format(String format, Object arg0)
       at MultiThreadedDbSeeder.Program.<Main>b__0(Int32 i) in C:\xxxx\Program.cs:line 22
       at System.Threading.Tasks.Parallel.<>c__DisplayClassf`1.b__c()
  InnerException:

Offending Line:
SqlCommand cmd = new SqlCommand(string.Format(@"insert into table_1 (vch_value) values('{1}')", "the value of i is " + i));

Solution:
Don’t forget that string.Format uses a zero based index.
Fixed Code:
SqlCommand cmd = new SqlCommand(string.Format(@"insert into table_1 (vch_value) values('{0}')", "the value of i is " + i));

Tuesday, March 29, 2011

The project file has been moved, renamed or is not on your computer

Solution:
1) Close the solution you have open.
2) In the project folder which is giving you problems, find the .suo file and delete it.
3) Reopen your solution and add the project back.

Saturday, March 26, 2011

Quartz.ObjectAlreadyExistsException: Unable to store Job with name: '' and group: 'DEFAULT', because one already exists with this identification.

Problem:
Exception Caught: Quartz.ObjectAlreadyExistsException: Unable to store Job with name: 'updateMyStuff' and group: 'DEFAULT', because one already exists with this identification.


Code:
                 // construct job info for every 10 seconds
                JobDetail = jobDetail = new JobDetail("updateMyStuff", null, typeof(UpdateMyOneMethod));
                trig = new CronTrigger();
                trig.CronExpression = new CronExpression("0/10 * * * * ?");
                trig.Name = " updateStuff ";
                sched.ScheduleJob(jobDetail, trig);

                // job for every day 12:00 am
                jobDetail = new JobDetail("updateMyStuff", null, typeof(UpdateMyOtherMethod));
                trig = new CronTrigger();
                trig.CronExpression = new CronExpression("0 0 0 * * ?");
                trig.Name = "updateStuff";
                sched.ScheduleJob(jobDetail, trig);



Solution:
I was clearly very tired and not paying attention and made a miserable copy/paste error.  Make sure the JobDetail has a unique name and same with the trigger!  Fixed Code, changes bolded:

                 // construct job info for every 10 seconds
                JobDetail = jobDetail = new JobDetail("updateMyStuff", null, typeof(UpdateMyOneMethod));
                trig = new CronTrigger();
                trig.CronExpression = new CronExpression("0/10 * * * * ?");
                trig.Name = " updateStuff ";
                sched.ScheduleJob(jobDetail, trig);

                // job for every day 12:00 am
                jobDetail = new JobDetail("updateMyOtherStuff", null, typeof(UpdateMyOtherMethod));
                trig = new CronTrigger();
                trig.CronExpression = new CronExpression("0 0 0 * * ?");
                trig.Name = "updateOtherStuff";
                sched.ScheduleJob(jobDetail, trig);



Sunday, March 20, 2011

Should I Create a Business Logic Layer?

The Business Logic Layer (BLL) is located between the Data Access Layer and the User Interface.  This is where your data is traslated from Database objects, like DataTables and DataRows into Classes and Objects.

Sometimes I use them, and other times I don’t.  This should become obvious in your design phase, when you learn whether or not your application is manipulating data after retrieving it from a database.  If your application is CRUD (Create Retrieve Update Delete), then most likely you don’t need a BLL.  It would be a waste of your time to add an unnecessary layer just because it is a “Proper Model”.

I find that with the Entity Framework the need for a business layer is further diminished.  The reason is that the auto generated code, that is the Model, takes out the need to translate DataRow to an Object.

Now, the only reason left to have a business layer is for business logic.  This is were much of your common code may go, like generation of reports, charts, and calculations.  The last few projects I have done only had partial business logic layers.  I only created them when necessary.

Saturday, March 19, 2011

The type or namespace name 'xxxx' does not exist in the namespace 'yyyy' (are you missing an assembly reference?)

Problem:
Error    1          The type or namespace name 'xxxx' does not exist in the namespace 'yyyy' (are you missing an assembly reference?)
Screenshot:

My environment: Visual Studio 2010, .Net 4.0.   Two projects involved, 1 is a dll, other is a console app.

You know your reference is correct, but you still get that reference error.  Your using clause fails, too.
Screenshot:
 

Solution:
In this case, my console app had the Target Framework in the project properties set to .NET Framework 4 Client Profile, the dll project had it set to .NET Framework 4.  I changed the console app to use .NET Framework 4, and everything built fine.

Console App Project properties before:

Console App Project properties after:


Thursday, March 17, 2011

Building Visual Studio Solutions Out of the Box

It is terrible when you are a developer at a new place and you open Visual Studio the first time, connect to your favorite source control like TFS or SVN, and open the desired solution, click build only to see many compile errors like invalid references.

Avoid creating this situation now by organizing your solutions so that they can easily be opened and built by new people.  You need to take into consideration how your solution is layed out, where external dependencies are, and how you present external tools.

First thing you need to do is make sure you solution file is at the root of all your projects.  If your all your projects are under c:/source, like in c:/source/MovieApp, then your solution should reside in c:/source.  You want to do this because it is not good practice to have a project reach up to a parent directory of the solution file into another branch for a referenced file or project.

Your structure should look like this:
C:/source
            MyMovieSolution.sln
C:/source/MovieApp
            MovieApp.csproj
C:/source/MvcMovie
            MvcMovie.csproj

Your next consideration needs to be references to external dependencies.  It is best if you have one location for this, right off the solution folder.  I usually call mine “External Dependencies”.  Then what you need to do is create an External Dependencies folder in the Visual Studio Solution and add your files from your “External Dependences” folder.  Now, whenever you reference a dll that is external, you use those files, and it will be gotten when you have a new developer get latest!  Easy enough.


















Next thing is to provide all your external tools in your solution.  I cannot tell you the number of times I had to go find an exact version of a tool because of some quirk that version had but no others did.  Check it in with your source, don’t store it on a shared network drive.  Update it?  Great, update it in your source control.
 









Last tip is do not set your output directories to the same folder!  Never do this, please.  Ever.  Leave them in bin.  If you are doing this, you most likely have an external dependency problem in one of your projects.  Fix that.

Saturday, March 5, 2011

The specified named connection is either not found in the configuration, not intended to be used with the EntityClient provider, or not valid.

Problem:

Server Error in '/' Application.


The specified named connection is either not found in the configuration, not intended to be used with the EntityClient provider, or not valid.

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. 

Exception Details: System.ArgumentException: The specified named connection is either not found in the configuration, not intended to be used with the EntityClient provider, or not valid.

Source Error: 
 
Line 32:         /// Initializes a new xxxxEntities object using the connection string found in the 'xxxxEntities' section of the application configuration file.
Line 33:         /// 
Line 34:         public xxxxEntities() : base("name=xxxxEntities", "xxxxEntities")
Line 35:         {
Line 36:             this.ContextOptions.LazyLoadingEnabled = true;

Source File: C:\Users\xxxx\Documents\Visual Studio 2010\Projects\xxxx.Designer.cs    Line: 34 

Stack Trace: 
 
[ArgumentException: The specified named connection is either not found in the configuration, not intended to be used with the EntityClient provider, or not valid.]
   System.Data.EntityClient.EntityConnection.ChangeConnectionString(String newConnectionString) +8080056
   System.Data.EntityClient.EntityConnection..ctor(String connectionString) +81
   System.Data.Objects.ObjectContext.CreateEntityConnection(String connectionString) +42
   System.Data.Objects.ObjectContext..ctor(String connectionString, String defaultContainerName) +16
   xxxx..ctor() in C:\Users\xxxxx.cs:34
   xxxx(Int32 id) in C:\Users\xxx.cs:25
   xxxx() in C:\Users\xxxx\Documents\Visual Studio 2010\Projects\xxxxDal.cs:36
   xxxx() in C:\Users\xxxx\Documents\Visual Studio 2010\Projects\xxxx.cs:13
   xxxx(Object sender, EventArgs e) in C:\Users\xxxxaspx.cs:145
   System.Web.UI.WebControls.Button.OnClick(EventArgs e) +118
   System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument) +112
   System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(String eventArgument) +10
   System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument) +13
   System.Web.UI.Page.RaisePostBackEvent(NameValueCollection postData) +36
   System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) +5563



Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.0.30319.1


Solution:
Add the entity connection string to the web.config file in the appropriate project.

Ambiguous reference - The type or namespace name 'Quartz' could not be found (are you missing a using directive or an assembly reference?)

Problem:
Environment: VS 2010 .Net 4.0 Console app using Quartz.Net http://quartznet.sourceforge.net/ as a Project Reference.

I would get an Ambiguous reference listing Quartz.IJob twice.

I had only 1 reference to Quarts in the list of References for my Project, that showed up as Quartz.2008.

Compile errors:
Error    1          The type or namespace name 'Quartz' could not be found (are you missing a using directive or an assembly reference?)
Error    5          The type or namespace name 'JobExecutionContext' could not be found (are you missing a using directive or an assembly reference?)  

Quartz Project properties had the target framework set to .NET Framework 3.5.

Solution:
Change the Target framework on the Quartz.2008 project properties to .NET Framework 4.  I ensured my project used .NET Framework 4, also.

If you get the following error in Quartz, then you had set the Target framework to .NET Framework Client Profile instead of .NET Framework 4.

Error    6          The type or namespace name 'HttpContext' could not be found (are you missing a using directive or an assembly reference?)          xxxx\Quartz.NET-1.0.3\src\Quartz\Util\LogicalThreadContext.cs    41        4            Quartz.2008




Failed to start monitoring changes to 'path' because access is denied.

Problem:
Brought up a web page and was presented with this error:

Server Error in '/' Application.


Failed to start monitoring changes to 'C:\Users\xxxx\AppData\Local\Temp\Temporary ASP.NET Files\root\b6230deb\7677a327\hash\hash.web' because access is denied.

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code. 

Exception Details: System.Web.HttpException: Failed to start monitoring changes to 'C:\Users\xxxx\AppData\Local\Temp\Temporary ASP.NET Files\root\b6230deb\7677a327\hash\hash.web' because access is denied.

Source Error: 
An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

Stack Trace: 
 
[HttpException (0x80070005): Failed to start monitoring changes to 'C:\Users\xxxx\AppData\Local\Temp\Temporary ASP.NET Files\root\b6230deb\7677a327\hash\hash.web' because access is denied.]
   System.Web.DirectoryMonitor.AddFileMonitor(String file) +8805891
   System.Web.DirectoryMonitor.StartMonitoringFileWithAssert(String file, FileChangeEventHandler callback, String alias) +94
   System.Web.FileChangesMonitor.StartMonitoringFile(String alias, FileChangeEventHandler callback) +340
   System.Web.Compilation.BuildManager.CheckTopLevelFilesUpToDate2(StandardDiskBuildResultCache diskCache) +790
   System.Web.Compilation.BuildManager.CheckTopLevelFilesUpToDate(StandardDiskBuildResultCache diskCache) +55
   System.Web.Compilation.BuildManager.RegularAppRuntimeModeInitialize() +174
   System.Web.Compilation.BuildManager.Initialize() +261
   System.Web.Compilation.BuildManager.InitializeBuildManager() +246
   System.Web.HttpRuntime.HostingInit(HostingEnvironmentFlags hostingFlags, PolicyLevel policyLevel, Exception appDomainCreationException) +350
 
[HttpException (0x80004005): Failed to start monitoring changes to 'C:\Users\xxxx\AppData\Local\Temp\Temporary ASP.NET Files\root\b6230deb\7677a327\hash\hash.web' because access is denied.]
   System.Web.HttpRuntime.FirstRequestInit(HttpContext context) +8950644
   System.Web.HttpRuntime.EnsureFirstRequestInit(HttpContext context) +97
   System.Web.HttpRuntime.ProcessRequestInternal(HttpWorkerRequest wr) +258



Version Information: Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.0.30319.1


Solution:
First, I had to close Visual Studio 2010, then I deleted c:\Users\xxxx\AppData\ocal\Temp\Temporary ASP.NET Files.
Reopened Visual Studio and ran it again in debug mode, worked fine.

Monday, February 28, 2011

MVC3 - Server Error in '/' Application. The resource cannot be found.



Problem:
MVC3 – You added your HttpPost method in your Controller, and you get:

Server Error in '/' Application.


The resource cannot be found.

Description: HTTP 404. The resource you are looking for (or one of its dependencies) could have been removed, had its name changed, or is temporarily unavailable.  Please review the following URL and make sure that it is spelled correctly. 

Requested URL: /Movies/Edit/1

Code:
        [HttpPost]
        public ActionResult Edit(Movie model)
        {
            try
            {
                var movie = db.Movies.Find(model.ID);

                UpdateModel(movie);
                db.SaveChanges();
                return RedirectToAction("Details", new {id=model.ID});
            }
            catch (Exception)
            {

                ModelState.AddModelError("", "Edit Failure, see inner exception");
            }

            return View(model);
        }



Solution:
You forgot to add the Get method to the Controller class:
Code:
        public ActionResult Edit(int id)
        {
            var movie = db.Movies.Find(id);
            if (movie == null)
                RedirectToAction("Index");
           
            return View(movie);
        }