Visuals are just the results of measurements and optionally may include some aggregation<\/p>\n\n\n\n
- Graphs<\/li>
- Charts<\/li>
- Charts of data totaled by month<\/li>
- Pie Charts<\/li><\/ul>\n\n\n\n
I gathered my thoughts and observations into this Miro Board<\/a> prior to starting development including some thoughts about GUI wireframes and SQL models. I think it’s always interesting to ideas evolve and looking back at this presentation, I only kept about 50% of the original ideas.<\/p>\n\n\n\n
Architecture<\/h2>\n\n\n\n
I’m interested in improving my skills with Python and Javascript so I constructed programs using a Python backend using the Django and Django Rest Framework and Javascript using React. <\/p>\n\n\n\n
The user creates and views measurements in one interface and visualizes the results in a second interface. The second interface stores the state of the users visualizations so that they can easily be recalled. A REST API is used to communicate with the database. When the user creates new measurements, a Django Signal is triggered that causes the results table to be updated.<\/p>\n\n\n
\n![\"\"](\"https:\/\/mechied.com\/wp-content\/uploads\/2022\/08\/Fulcrum-Concept-Update-8_3_2022-1.jpg\")
<\/figure>\n<\/div>\n\n\nHandling of Related Measurements<\/h3>\n\n\n\n
Creating a view of database state in React is not particularly novel. However allowing a user to create measurements that depend on other measurements required more cleverness and this section describes a little bit about how they are handled. <\/p>\n\n\n\n
The first step was defining how the frontend would communicate these measurements. This could have been a dedicated SQL database structure but to simplify implementation I decided to use a string syntax to define related measurements. Here is an example of the syntax:<\/p>\n\n\n\n
{p33m97l+20}[Test Project:Book Cost]*.25+{p33m104l+20}[Other Project:Marketing Cost]<\/p><\/blockquote>\n\n\n\n
The curly braces are what’s parsed by the database but the square brackets contain more human-friendly text descriptions of what project and measurement are being specified. Inside the curly braces there is a project tag, measurement tag, and time offset code. The project and measurement tags are actually the keys to the project and measurement in the database. The time offset code allows the result to “lag” or “lead” the measurement. Now since writing this by hand would be a pretty big ask of the users, they can actually select these values in a modal and insert them into the measurement. Users can specify addition, subtraction, multiplication, division, exponents, and parentheses (but more to come on this later).<\/p>\n\n\n
\n![\"\"](\"https:\/\/mechied.com\/wp-content\/uploads\/2022\/08\/image-500x335.png\")
<\/figure>\n<\/div>\n\n\nWhen a user saves a new measurement with this syntax a HTTP POST request is generated to the database. Django first creates a measurement configuration record:<\/p>\n\n\n\n
#views.py\n\nclass MeasureViewSet(ModelViewSet):\n serializer_class = MeasureSerializer\n\n def get_queryset(self):\n return Measure.objects.filter(project=self.kwargs['project_pk'])\n\n def get_serializer_context(self):\n return {'project_id': self.kwargs['project_pk'], \"request_data\": self.request.data}<\/code><\/pre>\n\n\n\nA measurement is described by several parameters which are a nested object inside the HTTP POST request. This requires a custom serializer to unpack and generate\/update the nested models.<\/p>\n\n\n\n
#serializers.py\n\nclass MeasureSerializer(serializers.ModelSerializer):\n\n def create(self, validated_data):\n parameter_data = validated_data.pop('parameters')\n project_id = self.context['project_id']\n\n measure = Measure.objects.create(\n project_id=project_id, **validated_data)\n\n for parameter in parameter_data:\n Parameter.objects.create(measure=measure, **parameter)\n return measure\n\n def update(self, instance, validated_data):\n\n if self.context['request_data'].get('parameters'):\n parameter_data = self.context['request_data'].pop('parameters')\n validated_parameter_data = validated_data.pop('parameters')\n # note that we're using raw data from the request data and not the validated data\n # when we go to update the nested object because we need the ID field which is stripped\n # from validated data\n\n parameter_dict = dict((i.id, i) for i in instance.parameters.all())\n\n for item_data in parameter_data:\n if 'id' in item_data:\n # if exists id remove from the dict and update\n parameter_item = parameter_dict.pop(item_data['id'])\n # remove id from validated data as we don't require it.\n item_data.pop('id')\n # loop through the rest of keys in validated data to assign it to its respective field\n for key in item_data.keys():\n setattr(parameter_item, key, item_data[key])\n\n parameter_item.save()\n else:\n # else create a new object\n Parameter.objects.create(measure=instance, **item_data)\n\n # delete remaining elements because they're not present in my update call\n if len(parameter_dict) > 0:\n for item in parameter_dict.values():\n item.delete()\n\n for attr, value in validated_data.items():\n setattr(instance, attr, value)\n\n instance.save()\n\n return instance\n\n parameters = ParameterSerializer(many=True, read_only=False)\n<\/code><\/pre>\n\n\n\nAfter the measurement is created, a signal function is called that uses the measurement’s parameters to perform calculations of the results and updates a table of measurement results. This function is fairly hefty so I won’t reproduce it here, but I will discuss a few interesting aspects.<\/p>\n\n\n\n
The first is what happens to parse the related expression string syntax and calculate the result. A series of regular expressions are used to pull the project, measure, and offset out of the expression string. Then the database queries for those results and puts the actual values into the expression string. Finally, since I’m avoiding writing my own parser, the python eval() method is used to act as the calculator (at some risk– I do pass an empty globals dictionary<\/a> but the user could write some stupid stuff into the string that I’m not checking for).<\/p>\n\n\n\n
Another key aspect is updated related expression measurements that depend on measurements that themselves have dependents. Suppose we have a measurement tree as follows:<\/p>\n\n\n
\n![\"\"](\"https:\/\/mechied.com\/wp-content\/uploads\/2022\/08\/Fulcrum-Concept-Frame-6-edited-500x354.jpg\")
<\/figure>\n<\/div>\n\n\nClearly the order of update will matter or else related measurements will use old data to generate. Before I wrote the function that handled this in the database I wrote a small python simulation and used the dependency scenario given in the diagram as test data. In writing it, I used dictionaries and functions like get_measure_by_id to make it clear where later I’d be using Django queries. My solution uses a tree data structure and recursion to cause the update to propagate through the tree in the correct order:<\/p>\n\n\n\n
class measure_list():\n def __init__(self) -> None: \n self.data = [measure(1, [2, 3, 4]),\n measure(2, [5]),\n measure(3, []),\n measure(4, []),\n measure(5, [3, 6, 7, 8]),\n measure(6, []),\n measure(7, []),\n measure(8, [3])]\n\n def get_measure_by_id(self, id:int):\n return list(filter(lambda x: x.get_id()==id,self.data))[0]\n\n def get_dependents_by_id(self,id:int) -> list[int]:\n return list(filter(lambda x: x.get_id()==id,self.data))[0].get_depends_on()\n\n def tell_id_to_update(self,id:int):\n measure = self.get_measure_by_id(id)\n did_measure_update = measure.update()\n return did_measure_update\n\nclass measure ():\n def __init__(self, id: int, depends_on: list[int], ) -> None:\n self.id = id\n self.depends_on = depends_on\n self.is_updated = False\n\n def set_measure_list(self,measure_list: measure_list) -> None:\n self.measure_list = measure_list\n\n def get_id(self) -> int:\n return self.id\n def get_depends_on(self) -> list[int]:\n return self.depends_on\n\n def update(self):\n if self.depends_on == []:\n self.is_updated = True\n print(str(self.get_id()) + \"has updated\")\n return True\n else:\n results = []\n for child in self.depends_on:\n results.append(self.measure_list.tell_id_to_update(child))\n self.is_updated = True\n print(str(self.get_id()) + \"has updated\")\n return True\n\nmy_measure_list = measure_list()\nfor measure in my_measure_list.data:\n measure.set_measure_list(my_measure_list)\n\nprint(my_measure_list.tell_id_to_update(1))<\/code><\/pre>\n\n\n\nOutput:\n3has updated\n6has updated\n7has updated\n3has updated\n8has updated\n5has updated\n2has updated\n3has updated\n4has updated\n1has updated\nTrue<\/code><\/pre>\n\n\n\nIf you look at the output the solution one inefficiency in this approach is that the child “leaf” nodes are told to update multiple times. It’s not harmful but it is slightly inefficient. I could store if the child has been updated and skip the update but in the actual Django implementation I didn’t want to have to manage the “child has been updated already on this update” state. So I’ve left it for now. I think it will be workable since the trees are not likely to be overly complex and the database updates are not particularly costly to perform.<\/p>\n\n\n\n
Another problem in this implementation is that circular dependency is not handled. I think that would be easy enough to detect and trigger an error so I plan on fixing it eventually.<\/p>\n\n\n\n
The code running on the database looks very similar to this prototype, just with functions like measure_query = Results.objects.filter(measure=measure_id).filter(date=date) used in place of the dictionary functions.<\/p>\n\n\n\n
One final big assumption in the code is that since a measurement could have results with null date, a single date, or several dates, a dependent measurement will first look for a child measurement at the same date, then at null date, then it will just assume 0.<\/p>\n\n\n\n
Summary<\/h2>\n\n\n\n
That was a whole lot of discussion of a backend to not really have much to show. I guess if it looked good it wouldn’t be the backend? Regardless I will post a chart to show that this actually works- I’m able to create, update, and delete measurements that then generate valid results in the database. In a future week I’ll share how the user actually makes these charts!<\/p>\n\n\n
\n![\"\"](\"https:\/\/mechied.com\/wp-content\/uploads\/2022\/08\/Screen-Shot-2022-08-03-at-3.46.51-PM-600x337.png\")
<\/figure>\n<\/div>","protected":false},"excerpt":{"rendered":"The main project I’m working on at The Recurse Center is an interactive project measurement dashboard. This week I’ll talk about the current state of the design and construction. I’m actively working on rearranging the user interface so a second post some other week will be needed to cover that. Motivation My wife, Justine, uses … Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"_vp_format_video_url":"","_vp_image_focal_point":[],"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[5],"tags":[],"class_list":["post-303","post","type-post","status-publish","format-standard","hentry","category-rc"],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/mechied.com\/index.php\/wp-json\/wp\/v2\/posts\/303","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/mechied.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mechied.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mechied.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mechied.com\/index.php\/wp-json\/wp\/v2\/comments?post=303"}],"version-history":[{"count":11,"href":"https:\/\/mechied.com\/index.php\/wp-json\/wp\/v2\/posts\/303\/revisions"}],"predecessor-version":[{"id":319,"href":"https:\/\/mechied.com\/index.php\/wp-json\/wp\/v2\/posts\/303\/revisions\/319"}],"wp:attachment":[{"href":"https:\/\/mechied.com\/index.php\/wp-json\/wp\/v2\/media?parent=303"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mechied.com\/index.php\/wp-json\/wp\/v2\/categories?post=303"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mechied.com\/index.php\/wp-json\/wp\/v2\/tags?post=303"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}